Sep 10 15:42:02.534129 ip-10-0-139-236 systemd[1]: Starting Kubernetes Kubelet... Sep 10 15:42:02.960798 ip-10-0-139-236 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 15:42:02.960798 ip-10-0-139-236 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 10 15:42:02.960798 ip-10-0-139-236 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 15:42:02.960798 ip-10-0-139-236 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 10 15:42:02.960798 ip-10-0-139-236 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 15:42:02.963640 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.963517 2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 10 15:42:02.969964 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.969944 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 15:42:02.969964 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.969962 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 15:42:02.969964 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.969966 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 15:42:02.969964 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.969969 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 15:42:02.969964 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.969972 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 15:42:02.970148 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.969975 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 15:42:02.970148 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.969978 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 15:42:02.970148 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.969981 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 15:42:02.970148 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.969985 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 15:42:02.970148 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.969989 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 15:42:02.970148 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.969992 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 15:42:02.970148 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.969995 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 15:42:02.970148 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.969999 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 15:42:02.970148 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970004 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 15:42:02.970148 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970007 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 15:42:02.970148 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970010 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 15:42:02.970148 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970013 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 15:42:02.970148 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970015 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 15:42:02.970148 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970018 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 15:42:02.970148 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970021 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 15:42:02.970148 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970024 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 15:42:02.970148 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970027 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 15:42:02.970148 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970029 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 15:42:02.970148 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970032 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 15:42:02.970610 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970034 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 15:42:02.970610 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970037 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 15:42:02.970610 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970043 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 15:42:02.970610 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970045 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 15:42:02.970610 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970048 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 15:42:02.970610 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970050 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 15:42:02.970610 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970054 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 15:42:02.970610 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970057 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 15:42:02.970610 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970060 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 15:42:02.970610 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970063 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 15:42:02.970610 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970066 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 15:42:02.970610 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970069 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 15:42:02.970610 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970071 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 15:42:02.970610 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970074 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 15:42:02.970610 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970077 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 15:42:02.970610 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970079 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 15:42:02.970610 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970082 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 15:42:02.970610 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970085 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 15:42:02.970610 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970087 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 15:42:02.971143 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970090 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 15:42:02.971143 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970093 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 15:42:02.971143 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970096 2576 feature_gate.go:328] unrecognized feature gate: Example Sep 10 15:42:02.971143 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970099 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 15:42:02.971143 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970101 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 15:42:02.971143 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970103 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 15:42:02.971143 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970106 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 15:42:02.971143 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970108 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 15:42:02.971143 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970111 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 15:42:02.971143 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970113 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 15:42:02.971143 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970116 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 15:42:02.971143 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970118 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 15:42:02.971143 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970121 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 15:42:02.971143 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970123 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 15:42:02.971143 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970126 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 15:42:02.971143 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970129 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 15:42:02.971143 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970131 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 15:42:02.971143 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970134 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 15:42:02.971143 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970136 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 15:42:02.971143 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970139 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 15:42:02.971633 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970141 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 15:42:02.971633 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970144 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 15:42:02.971633 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970146 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 15:42:02.971633 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970149 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 15:42:02.971633 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970151 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 15:42:02.971633 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970154 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 15:42:02.971633 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970157 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 15:42:02.971633 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970160 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 15:42:02.971633 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970163 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 15:42:02.971633 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970165 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 15:42:02.971633 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970168 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 15:42:02.971633 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970170 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 15:42:02.971633 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970173 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 15:42:02.971633 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970175 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 15:42:02.971633 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970178 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 15:42:02.971633 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970182 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 15:42:02.971633 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970186 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 15:42:02.971633 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970189 2576 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 15:42:02.971633 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970191 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 15:42:02.972127 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970194 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 15:42:02.972127 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970196 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 15:42:02.972127 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970199 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 15:42:02.972127 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970202 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 15:42:02.972127 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970205 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 15:42:02.972127 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970207 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 15:42:02.972127 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970210 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 15:42:02.972127 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970213 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 15:42:02.972127 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970215 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 15:42:02.972127 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970219 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 15:42:02.972127 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970221 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 15:42:02.972127 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970224 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 15:42:02.972127 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970226 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 15:42:02.972127 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970229 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 15:42:02.972127 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970231 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 15:42:02.972127 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970234 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 15:42:02.972127 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970236 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 15:42:02.972127 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970966 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 15:42:02.972127 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970973 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 15:42:02.972127 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970976 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 15:42:02.972622 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970980 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 15:42:02.972622 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970983 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 15:42:02.972622 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970987 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 15:42:02.972622 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970991 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 15:42:02.972622 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970994 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 15:42:02.972622 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970997 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 15:42:02.972622 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.970999 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 15:42:02.972622 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971002 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 15:42:02.972622 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971004 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 15:42:02.972622 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971007 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 15:42:02.972622 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971011 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 15:42:02.972622 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971015 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 15:42:02.972622 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971018 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 15:42:02.972622 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971021 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 15:42:02.972622 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971024 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 15:42:02.972622 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971026 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 15:42:02.972622 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971030 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 15:42:02.972622 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971032 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 15:42:02.972622 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971035 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 15:42:02.973125 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971037 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 15:42:02.973125 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971040 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 15:42:02.973125 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971042 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 15:42:02.973125 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971045 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 15:42:02.973125 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971048 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 15:42:02.973125 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971051 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 15:42:02.973125 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971053 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 15:42:02.973125 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971056 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 15:42:02.973125 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971059 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 15:42:02.973125 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971061 2576 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 15:42:02.973125 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971064 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 15:42:02.973125 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971066 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 15:42:02.973125 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971069 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 15:42:02.973125 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971072 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 15:42:02.973125 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971075 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 15:42:02.973125 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971077 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 15:42:02.973125 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971080 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 15:42:02.973125 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971082 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 15:42:02.973125 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971085 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 15:42:02.973125 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971087 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 15:42:02.973779 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971090 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 15:42:02.973779 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971092 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 15:42:02.973779 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971094 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 15:42:02.973779 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971097 2576 feature_gate.go:328] unrecognized feature gate: Example Sep 10 15:42:02.973779 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971100 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 15:42:02.973779 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971102 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 15:42:02.973779 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971105 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 15:42:02.973779 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971107 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 15:42:02.973779 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971109 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 15:42:02.973779 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971112 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 15:42:02.973779 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971114 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 15:42:02.973779 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971116 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 15:42:02.973779 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971119 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 15:42:02.973779 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971121 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 15:42:02.973779 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971124 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 15:42:02.973779 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971126 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 15:42:02.973779 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971129 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 15:42:02.973779 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971131 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 15:42:02.973779 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971133 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 15:42:02.973779 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971136 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 15:42:02.974278 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971138 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 15:42:02.974278 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971141 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 15:42:02.974278 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971143 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 15:42:02.974278 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971146 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 15:42:02.974278 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971148 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 15:42:02.974278 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971152 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 15:42:02.974278 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971155 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 15:42:02.974278 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971158 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 15:42:02.974278 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971160 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 15:42:02.974278 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971163 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 15:42:02.974278 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971165 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 15:42:02.974278 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971168 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 15:42:02.974278 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971170 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 15:42:02.974278 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971173 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 15:42:02.974278 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971175 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 15:42:02.974278 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971178 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 15:42:02.974278 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971181 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 15:42:02.974278 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971184 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 15:42:02.974278 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971188 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 15:42:02.974766 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971191 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 15:42:02.974766 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971194 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 15:42:02.974766 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971197 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 15:42:02.974766 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971200 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 15:42:02.974766 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971203 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 15:42:02.974766 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971206 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 15:42:02.974766 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971209 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 15:42:02.974766 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971211 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 15:42:02.974766 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971213 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 15:42:02.974766 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971216 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 15:42:02.974766 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971219 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 15:42:02.974766 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971221 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 15:42:02.974766 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971224 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 15:42:02.974766 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971226 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 15:42:02.974766 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971229 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 15:42:02.974766 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971231 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 15:42:02.974766 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971234 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 15:42:02.974766 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971237 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 15:42:02.974766 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971630 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 15:42:02.974766 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971635 2576 feature_gate.go:328] unrecognized feature gate: Example Sep 10 15:42:02.975263 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971638 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 15:42:02.975263 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971641 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 15:42:02.975263 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971644 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 15:42:02.975263 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971647 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 15:42:02.975263 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971650 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 15:42:02.975263 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971653 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 15:42:02.975263 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971655 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 15:42:02.975263 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971658 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 15:42:02.975263 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971661 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 15:42:02.975263 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971663 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 15:42:02.975263 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971666 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 15:42:02.975263 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971668 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 15:42:02.975263 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971671 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 15:42:02.975263 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971673 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 15:42:02.975263 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971675 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 15:42:02.975263 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971678 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 15:42:02.975263 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971681 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 15:42:02.975263 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971683 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 15:42:02.975263 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971685 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 15:42:02.975263 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971688 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 15:42:02.975785 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971690 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 15:42:02.975785 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971693 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 15:42:02.975785 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971696 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 15:42:02.975785 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971698 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 15:42:02.975785 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971701 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 15:42:02.975785 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971703 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 15:42:02.975785 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971706 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 15:42:02.975785 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971708 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 15:42:02.975785 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971710 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 15:42:02.975785 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971716 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 15:42:02.975785 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971719 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 15:42:02.975785 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971723 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 15:42:02.975785 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971726 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 15:42:02.975785 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971744 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 15:42:02.975785 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971747 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 15:42:02.975785 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971750 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 15:42:02.975785 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971753 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 15:42:02.975785 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971756 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 15:42:02.975785 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971758 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 15:42:02.976260 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971761 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 15:42:02.976260 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971764 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 15:42:02.976260 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971766 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 15:42:02.976260 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971769 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 15:42:02.976260 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971773 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 15:42:02.976260 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971777 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 15:42:02.976260 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971781 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 15:42:02.976260 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971784 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 15:42:02.976260 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971788 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 15:42:02.976260 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971791 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 15:42:02.976260 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971794 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 15:42:02.976260 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971796 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 15:42:02.976260 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971799 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 15:42:02.976260 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971801 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 15:42:02.976260 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971804 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 15:42:02.976260 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971806 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 15:42:02.976260 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971810 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 15:42:02.976260 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971812 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 15:42:02.976260 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971815 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 15:42:02.976756 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971817 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 15:42:02.976756 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971820 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 15:42:02.976756 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971822 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 15:42:02.976756 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971826 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 15:42:02.976756 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971829 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 15:42:02.976756 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971831 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 15:42:02.976756 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971834 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 15:42:02.976756 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971836 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 15:42:02.976756 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971839 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 15:42:02.976756 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971842 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 15:42:02.976756 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971845 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 15:42:02.976756 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971847 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 15:42:02.976756 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971850 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 15:42:02.976756 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971852 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 15:42:02.976756 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971855 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 15:42:02.976756 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971857 2576 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 15:42:02.976756 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971860 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 15:42:02.976756 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971862 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 15:42:02.976756 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971865 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 15:42:02.976756 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971867 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 15:42:02.977258 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971870 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 15:42:02.977258 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971873 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 15:42:02.977258 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971876 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 15:42:02.977258 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971878 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 15:42:02.977258 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971881 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 15:42:02.977258 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971883 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 15:42:02.977258 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971886 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 15:42:02.977258 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971888 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 15:42:02.977258 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971891 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 15:42:02.977258 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971893 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 15:42:02.977258 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971896 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 15:42:02.977258 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971898 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 15:42:02.977258 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971901 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 15:42:02.977258 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971903 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 15:42:02.977258 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971906 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 15:42:02.977258 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971909 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 15:42:02.977258 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971912 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 15:42:02.977258 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971914 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 15:42:02.977258 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.971917 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 15:42:02.977258 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.971996 2576 flags.go:64] FLAG: --address="0.0.0.0" Sep 10 15:42:02.977758 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972038 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 10 15:42:02.977758 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972049 2576 flags.go:64] FLAG: --anonymous-auth="true" Sep 10 15:42:02.977758 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972054 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 10 15:42:02.977758 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972058 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 10 15:42:02.977758 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972061 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 10 15:42:02.977758 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972065 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 10 15:42:02.977758 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972070 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 10 15:42:02.977758 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972073 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 10 15:42:02.977758 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972076 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 10 15:42:02.977758 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972079 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 10 15:42:02.977758 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972083 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 10 15:42:02.977758 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972086 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 10 15:42:02.977758 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972088 2576 flags.go:64] FLAG: --cgroup-root="" Sep 10 15:42:02.977758 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972091 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 10 15:42:02.977758 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972094 2576 flags.go:64] FLAG: --client-ca-file="" Sep 10 15:42:02.977758 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972097 2576 flags.go:64] FLAG: --cloud-provider="external" Sep 10 15:42:02.977758 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972099 2576 flags.go:64] FLAG: --cluster-dns="[]" Sep 10 15:42:02.977758 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972104 2576 flags.go:64] FLAG: --cluster-domain="" Sep 10 15:42:02.977758 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972107 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 10 15:42:02.977758 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972110 2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 10 15:42:02.977758 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972113 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 10 15:42:02.977758 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972116 2576 flags.go:64] FLAG: --container-log-max-files="5" Sep 10 15:42:02.977758 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972119 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 10 15:42:02.977758 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972124 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 10 15:42:02.978352 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972127 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 10 15:42:02.978352 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972130 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 10 15:42:02.978352 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972133 2576 flags.go:64] FLAG: --contention-profiling="false" Sep 10 15:42:02.978352 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972138 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 10 15:42:02.978352 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972141 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 10 15:42:02.978352 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972144 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 10 15:42:02.978352 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972147 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 10 15:42:02.978352 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972151 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 10 15:42:02.978352 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972154 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 10 15:42:02.978352 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972157 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 10 15:42:02.978352 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972159 2576 flags.go:64] FLAG: --enable-load-reader="false" Sep 10 15:42:02.978352 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972162 2576 flags.go:64] FLAG: --enable-server="true" Sep 10 15:42:02.978352 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972165 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 10 15:42:02.978352 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972170 2576 flags.go:64] FLAG: --event-burst="100" Sep 10 15:42:02.978352 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972172 2576 flags.go:64] FLAG: --event-qps="50" Sep 10 15:42:02.978352 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972175 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 10 15:42:02.978352 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972178 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 10 15:42:02.978352 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972181 2576 flags.go:64] FLAG: --eviction-hard="" Sep 10 15:42:02.978352 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972184 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 10 15:42:02.978352 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972187 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 10 15:42:02.978352 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972190 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 10 15:42:02.978352 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972193 2576 flags.go:64] FLAG: --eviction-soft="" Sep 10 15:42:02.978352 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972196 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 10 15:42:02.978352 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972199 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 10 15:42:02.978352 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972202 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 10 15:42:02.979041 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972204 2576 flags.go:64] FLAG: --experimental-mounter-path="" Sep 10 15:42:02.979041 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972207 2576 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 10 15:42:02.979041 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972210 2576 flags.go:64] FLAG: --fail-swap-on="true" Sep 10 15:42:02.979041 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972213 2576 flags.go:64] FLAG: --feature-gates="" Sep 10 15:42:02.979041 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972217 2576 flags.go:64] FLAG: --file-check-frequency="20s" Sep 10 15:42:02.979041 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972219 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 10 15:42:02.979041 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972223 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 10 15:42:02.979041 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972226 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 10 15:42:02.979041 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972229 2576 flags.go:64] FLAG: --healthz-port="10248" Sep 10 15:42:02.979041 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972232 2576 flags.go:64] FLAG: --help="false" Sep 10 15:42:02.979041 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972235 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-139-236.ec2.internal" Sep 10 15:42:02.979041 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972239 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 10 15:42:02.979041 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972241 2576 flags.go:64] FLAG: --http-check-frequency="20s" Sep 10 15:42:02.979041 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972244 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 10 15:42:02.979041 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972247 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 10 15:42:02.979041 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972251 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 10 15:42:02.979041 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972253 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 10 15:42:02.979041 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972256 2576 flags.go:64] FLAG: --image-service-endpoint="" Sep 10 15:42:02.979041 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972259 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 10 15:42:02.979041 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972262 2576 flags.go:64] FLAG: --kube-api-burst="100" Sep 10 15:42:02.979041 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972264 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 10 15:42:02.979041 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972267 2576 flags.go:64] FLAG: --kube-api-qps="50" Sep 10 15:42:02.979041 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972270 2576 flags.go:64] FLAG: --kube-reserved="" Sep 10 15:42:02.979041 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972273 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 10 15:42:02.979646 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972275 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 10 15:42:02.979646 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972278 2576 flags.go:64] FLAG: --kubelet-cgroups="" Sep 10 15:42:02.979646 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972281 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 10 15:42:02.979646 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972284 2576 flags.go:64] FLAG: --lock-file="" Sep 10 15:42:02.979646 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972286 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 10 15:42:02.979646 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972289 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 10 15:42:02.979646 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972292 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 10 15:42:02.979646 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972297 2576 flags.go:64] FLAG: --log-json-split-stream="false" Sep 10 15:42:02.979646 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972300 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 10 15:42:02.979646 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972302 2576 flags.go:64] FLAG: --log-text-split-stream="false" Sep 10 15:42:02.979646 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972305 2576 flags.go:64] FLAG: --logging-format="text" Sep 10 15:42:02.979646 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972308 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 10 15:42:02.979646 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972311 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 10 15:42:02.979646 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972313 2576 flags.go:64] FLAG: --manifest-url="" Sep 10 15:42:02.979646 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972316 2576 flags.go:64] FLAG: --manifest-url-header="" Sep 10 15:42:02.979646 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972320 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 10 15:42:02.979646 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972323 2576 flags.go:64] FLAG: --max-open-files="1000000" Sep 10 15:42:02.979646 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972327 2576 flags.go:64] FLAG: --max-pods="110" Sep 10 15:42:02.979646 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972330 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 10 15:42:02.979646 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972334 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 10 15:42:02.979646 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972336 2576 flags.go:64] FLAG: --memory-manager-policy="None" Sep 10 15:42:02.979646 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972339 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 10 15:42:02.979646 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972342 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 10 15:42:02.979646 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972345 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 10 15:42:02.979646 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972348 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 10 15:42:02.980295 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972354 2576 flags.go:64] FLAG: --node-status-max-images="50" Sep 10 15:42:02.980295 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972357 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 10 15:42:02.980295 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972360 2576 flags.go:64] FLAG: --oom-score-adj="-999" Sep 10 15:42:02.980295 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972364 2576 flags.go:64] FLAG: --pod-cidr="" Sep 10 15:42:02.980295 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972366 2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 10 15:42:02.980295 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972372 2576 flags.go:64] FLAG: --pod-manifest-path="" Sep 10 15:42:02.980295 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972374 2576 flags.go:64] FLAG: --pod-max-pids="-1" Sep 10 15:42:02.980295 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972377 2576 flags.go:64] FLAG: --pods-per-core="0" Sep 10 15:42:02.980295 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972380 2576 flags.go:64] FLAG: --port="10250" Sep 10 15:42:02.980295 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972383 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 10 15:42:02.980295 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972386 2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-04f111381f3fd8f21" Sep 10 15:42:02.980295 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972389 2576 flags.go:64] FLAG: --qos-reserved="" Sep 10 15:42:02.980295 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972392 2576 flags.go:64] FLAG: --read-only-port="10255" Sep 10 15:42:02.980295 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972394 2576 flags.go:64] FLAG: --register-node="true" Sep 10 15:42:02.980295 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972397 2576 flags.go:64] FLAG: --register-with-taints="" Sep 10 15:42:02.980295 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972401 2576 flags.go:64] FLAG: --registry-burst="10" Sep 10 15:42:02.980295 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972403 2576 flags.go:64] FLAG: --registry-qps="5" Sep 10 15:42:02.980295 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972406 2576 flags.go:64] FLAG: --reserved-cpus="" Sep 10 15:42:02.980295 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972408 2576 flags.go:64] FLAG: --reserved-memory="" Sep 10 15:42:02.980295 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972412 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 10 15:42:02.980295 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972415 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 10 15:42:02.980295 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972418 2576 flags.go:64] FLAG: --rotate-certificates="false" Sep 10 15:42:02.980295 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972420 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 10 15:42:02.980295 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972423 2576 flags.go:64] FLAG: --runonce="false" Sep 10 15:42:02.980295 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972426 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 10 15:42:02.980950 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972429 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 10 15:42:02.980950 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972432 2576 flags.go:64] FLAG: --seccomp-default="false" Sep 10 15:42:02.980950 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972434 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 10 15:42:02.980950 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972444 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 10 15:42:02.980950 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972448 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 10 15:42:02.980950 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972451 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 10 15:42:02.980950 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972454 2576 flags.go:64] FLAG: --storage-driver-password="root" Sep 10 15:42:02.980950 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972457 2576 flags.go:64] FLAG: --storage-driver-secure="false" Sep 10 15:42:02.980950 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972460 2576 flags.go:64] FLAG: --storage-driver-table="stats" Sep 10 15:42:02.980950 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972464 2576 flags.go:64] FLAG: --storage-driver-user="root" Sep 10 15:42:02.980950 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972467 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 10 15:42:02.980950 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972470 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 10 15:42:02.980950 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972472 2576 flags.go:64] FLAG: --system-cgroups="" Sep 10 15:42:02.980950 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972475 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 10 15:42:02.980950 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972480 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 10 15:42:02.980950 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972483 2576 flags.go:64] FLAG: --tls-cert-file="" Sep 10 15:42:02.980950 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972486 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 10 15:42:02.980950 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972490 2576 flags.go:64] FLAG: --tls-min-version="" Sep 10 15:42:02.980950 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972493 2576 flags.go:64] FLAG: --tls-private-key-file="" Sep 10 15:42:02.980950 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972495 2576 flags.go:64] FLAG: --topology-manager-policy="none" Sep 10 15:42:02.980950 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972498 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 10 15:42:02.980950 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972500 2576 flags.go:64] FLAG: --topology-manager-scope="container" Sep 10 15:42:02.980950 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972503 2576 flags.go:64] FLAG: --v="2" Sep 10 15:42:02.980950 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972507 2576 flags.go:64] FLAG: --version="false" Sep 10 15:42:02.980950 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972511 2576 flags.go:64] FLAG: --vmodule="" Sep 10 15:42:02.981578 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972514 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 10 15:42:02.981578 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.972517 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 10 15:42:02.981578 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972649 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 15:42:02.981578 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972654 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 15:42:02.981578 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972657 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 15:42:02.981578 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972660 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 15:42:02.981578 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972663 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 15:42:02.981578 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972666 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 15:42:02.981578 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972669 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 15:42:02.981578 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972671 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 15:42:02.981578 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972674 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 15:42:02.981578 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972676 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 15:42:02.981578 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972679 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 15:42:02.981578 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972681 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 15:42:02.981578 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972684 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 15:42:02.981578 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972686 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 15:42:02.981578 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972690 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 15:42:02.981578 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972693 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 15:42:02.981578 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972695 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 15:42:02.981578 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972697 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 15:42:02.982099 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972700 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 15:42:02.982099 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972702 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 15:42:02.982099 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972705 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 15:42:02.982099 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972707 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 15:42:02.982099 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972710 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 15:42:02.982099 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972712 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 15:42:02.982099 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972714 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 15:42:02.982099 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972717 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 15:42:02.982099 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972719 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 15:42:02.982099 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972722 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 15:42:02.982099 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972724 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 15:42:02.982099 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972726 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 15:42:02.982099 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972742 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 15:42:02.982099 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972744 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 15:42:02.982099 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972747 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 15:42:02.982099 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972749 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 15:42:02.982099 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972752 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 15:42:02.982099 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972755 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 15:42:02.982099 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972757 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 15:42:02.982099 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972760 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 15:42:02.982927 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972764 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 15:42:02.982927 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972767 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 15:42:02.982927 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972769 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 15:42:02.982927 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972772 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 15:42:02.982927 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972775 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 15:42:02.982927 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972777 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 15:42:02.982927 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972780 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 15:42:02.982927 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972782 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 15:42:02.982927 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972786 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 15:42:02.982927 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972789 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 15:42:02.982927 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972791 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 15:42:02.982927 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972794 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 15:42:02.982927 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972796 2576 feature_gate.go:328] unrecognized feature gate: Example Sep 10 15:42:02.982927 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972799 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 15:42:02.982927 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972801 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 15:42:02.982927 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972804 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 15:42:02.982927 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972807 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 15:42:02.982927 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972809 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 15:42:02.982927 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972812 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 15:42:02.982927 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972815 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 15:42:02.983480 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972817 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 15:42:02.983480 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972820 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 15:42:02.983480 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972824 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 15:42:02.983480 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972827 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 15:42:02.983480 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972831 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 15:42:02.983480 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972835 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 15:42:02.983480 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972838 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 15:42:02.983480 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972840 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 15:42:02.983480 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972843 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 15:42:02.983480 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972846 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 15:42:02.983480 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972849 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 15:42:02.983480 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972851 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 15:42:02.983480 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972854 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 15:42:02.983480 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972858 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 15:42:02.983480 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972860 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 15:42:02.983480 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972863 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 15:42:02.983480 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972866 2576 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 15:42:02.983480 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972868 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 15:42:02.983480 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972871 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 15:42:02.983986 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972873 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 15:42:02.983986 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972878 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 15:42:02.983986 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972882 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 15:42:02.983986 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972884 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 15:42:02.983986 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972887 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 15:42:02.983986 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972890 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 15:42:02.983986 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972893 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 15:42:02.983986 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972895 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 15:42:02.983986 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972898 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 15:42:02.983986 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972900 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 15:42:02.983986 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972903 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 15:42:02.983986 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972905 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 15:42:02.983986 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972908 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 15:42:02.983986 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972910 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 15:42:02.983986 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972913 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 15:42:02.983986 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972916 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 15:42:02.983986 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972918 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 15:42:02.983986 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972921 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 15:42:02.983986 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972923 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 15:42:02.984470 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972925 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 15:42:02.984470 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972928 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 15:42:02.984470 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.972930 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 15:42:02.984470 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.973676 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 15:42:02.984470 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.981759 2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 10 15:42:02.984470 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.981774 2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 10 15:42:02.984470 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.981792 2576 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 10 15:42:02.984470 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.981798 2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 10 15:42:02.984470 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.981860 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 15:42:02.984470 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.981865 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 15:42:02.984470 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.981869 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 15:42:02.984470 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.981871 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 15:42:02.984470 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.981874 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 15:42:02.984470 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.981877 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 15:42:02.984470 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.981880 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 15:42:02.984882 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.981883 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 15:42:02.984882 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.981886 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 15:42:02.984882 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.981933 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 15:42:02.984882 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.981997 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 15:42:02.984882 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.982001 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 15:42:02.984882 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.982004 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 15:42:02.984882 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.982007 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 15:42:02.984882 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.982010 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 15:42:02.984882 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.982014 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 15:42:02.984882 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.982019 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 15:42:02.984882 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.982022 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 15:42:02.984882 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.982025 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 15:42:02.984882 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.982029 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 15:42:02.984882 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.982033 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 15:42:02.984882 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.982038 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 15:42:02.984882 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.982047 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 15:42:02.984882 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.982052 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 15:42:02.984882 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.982056 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 15:42:02.984882 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983347 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 15:42:02.985346 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983381 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 15:42:02.985346 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983405 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 15:42:02.985346 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983413 2576 feature_gate.go:328] unrecognized feature gate: Example Sep 10 15:42:02.985346 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983419 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 15:42:02.985346 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983425 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 15:42:02.985346 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983430 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 15:42:02.985346 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983435 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 15:42:02.985346 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983449 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 15:42:02.985346 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983590 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 15:42:02.985346 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983602 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 15:42:02.985346 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983606 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 15:42:02.985346 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983610 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 15:42:02.985346 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983616 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 15:42:02.985346 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983620 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 15:42:02.985346 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983623 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 15:42:02.985346 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983625 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 15:42:02.985346 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983628 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 15:42:02.985346 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983631 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 15:42:02.985346 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983634 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 15:42:02.985346 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983636 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 15:42:02.985902 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983639 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 15:42:02.985902 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983641 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 15:42:02.985902 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983644 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 15:42:02.985902 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983646 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 15:42:02.985902 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983650 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 15:42:02.985902 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983652 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 15:42:02.985902 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983655 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 15:42:02.985902 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983657 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 15:42:02.985902 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983660 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 15:42:02.985902 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983663 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 15:42:02.985902 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983665 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 15:42:02.985902 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983668 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 15:42:02.985902 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983670 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 15:42:02.985902 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983673 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 15:42:02.985902 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983675 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 15:42:02.985902 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983678 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 15:42:02.985902 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983681 2576 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 15:42:02.985902 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983683 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 15:42:02.985902 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983686 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 15:42:02.985902 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983688 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 15:42:02.986397 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983693 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 15:42:02.986397 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983698 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 15:42:02.986397 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983701 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 15:42:02.986397 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983703 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 15:42:02.986397 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983706 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 15:42:02.986397 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983709 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 15:42:02.986397 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983712 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 15:42:02.986397 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983714 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 15:42:02.986397 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983717 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 15:42:02.986397 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983720 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 15:42:02.986397 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983723 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 15:42:02.986397 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983725 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 15:42:02.986397 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983746 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 15:42:02.986397 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983750 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 15:42:02.986397 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983753 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 15:42:02.986397 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983755 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 15:42:02.986397 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983758 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 15:42:02.986397 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983760 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 15:42:02.986397 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983763 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 15:42:02.986905 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983765 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 15:42:02.986905 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983768 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 15:42:02.986905 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983770 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 15:42:02.986905 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983773 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 15:42:02.986905 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983775 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 15:42:02.986905 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983778 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 15:42:02.986905 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983781 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 15:42:02.986905 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983784 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 15:42:02.986905 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983787 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 15:42:02.986905 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983789 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 15:42:02.986905 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983792 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 15:42:02.986905 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983795 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 15:42:02.986905 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983797 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 15:42:02.986905 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983800 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 15:42:02.986905 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.983805 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 15:42:02.987274 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983944 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 15:42:02.987274 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983948 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 15:42:02.987274 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983951 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 15:42:02.987274 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983954 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 15:42:02.987274 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983957 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 15:42:02.987274 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983960 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 15:42:02.987274 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983963 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 15:42:02.987274 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983965 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 15:42:02.987274 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983968 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 15:42:02.987274 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983971 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 15:42:02.987274 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983973 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 15:42:02.987274 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983976 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 15:42:02.987274 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983978 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 15:42:02.987274 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983981 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 15:42:02.987274 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983983 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 15:42:02.987274 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983986 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 15:42:02.987274 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983988 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 15:42:02.987274 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983991 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 15:42:02.987274 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983993 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 15:42:02.987274 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983996 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 15:42:02.987773 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.983998 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 15:42:02.987773 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984001 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 15:42:02.987773 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984004 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 15:42:02.987773 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984007 2576 feature_gate.go:328] unrecognized feature gate: Example Sep 10 15:42:02.987773 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984010 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 15:42:02.987773 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984012 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 15:42:02.987773 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984015 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 15:42:02.987773 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984017 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 15:42:02.987773 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984020 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 15:42:02.987773 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984022 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 15:42:02.987773 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984025 2576 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 15:42:02.987773 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984027 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 15:42:02.987773 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984030 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 15:42:02.987773 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984032 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 15:42:02.987773 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984037 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 15:42:02.987773 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984040 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 15:42:02.987773 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984043 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 15:42:02.987773 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984046 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 15:42:02.987773 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984049 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 15:42:02.988289 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984051 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 15:42:02.988289 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984054 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 15:42:02.988289 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984057 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 15:42:02.988289 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984060 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 15:42:02.988289 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984062 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 15:42:02.988289 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984065 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 15:42:02.988289 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984067 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 15:42:02.988289 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984069 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 15:42:02.988289 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984072 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 15:42:02.988289 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984074 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 15:42:02.988289 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984077 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 15:42:02.988289 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984079 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 15:42:02.988289 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984082 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 15:42:02.988289 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984084 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 15:42:02.988289 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984087 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 15:42:02.988289 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984089 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 15:42:02.988289 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984092 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 15:42:02.988289 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984094 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 15:42:02.988289 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984097 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 15:42:02.988289 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984099 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 15:42:02.988812 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984102 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 15:42:02.988812 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984104 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 15:42:02.988812 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984106 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 15:42:02.988812 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984109 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 15:42:02.988812 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984112 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 15:42:02.988812 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984114 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 15:42:02.988812 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984116 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 15:42:02.988812 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984119 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 15:42:02.988812 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984122 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 15:42:02.988812 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984124 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 15:42:02.988812 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984127 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 15:42:02.988812 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984129 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 15:42:02.988812 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984132 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 15:42:02.988812 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984134 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 15:42:02.988812 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984137 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 15:42:02.988812 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984139 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 15:42:02.988812 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984141 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 15:42:02.988812 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984144 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 15:42:02.988812 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984146 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 15:42:02.988812 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984148 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 15:42:02.989313 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984151 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 15:42:02.989313 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984153 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 15:42:02.989313 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984157 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 15:42:02.989313 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984160 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 15:42:02.989313 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984163 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 15:42:02.989313 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984166 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 15:42:02.989313 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984168 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 15:42:02.989313 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984171 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 15:42:02.989313 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984173 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 15:42:02.989313 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984176 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 15:42:02.989313 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984179 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 15:42:02.989313 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984181 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 15:42:02.989313 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984185 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 15:42:02.989313 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984188 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 15:42:02.989313 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984191 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 15:42:02.989313 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984194 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 15:42:02.989313 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984197 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 15:42:02.989313 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984200 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 15:42:02.989313 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984202 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 15:42:02.989829 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:02.984205 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 15:42:02.989829 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.984209 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 15:42:02.989829 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.984312 2576 server.go:962] "Client rotation is on, will bootstrap in background" Sep 10 15:42:02.989829 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.987832 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 10 15:42:02.989829 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.988946 2576 server.go:1019] "Starting client certificate rotation" Sep 10 15:42:02.989829 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.989044 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 10 15:42:02.989829 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.989081 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 10 15:42:02.991747 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.991720 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 10 15:42:02.992887 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:02.992856 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 10 15:42:03.006794 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.006775 2576 log.go:25] "Validated CRI v1 runtime API" Sep 10 15:42:03.011141 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.011125 2576 log.go:25] "Validated CRI v1 image API" Sep 10 15:42:03.012093 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.012072 2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 10 15:42:03.017081 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.017063 2576 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 cb12c1b2-b15f-4115-82d1-db339980c886:/dev/nvme0n1p4 ec3e38c3-9585-4fcd-a03c-65bfb1fd8786:/dev/nvme0n1p3] Sep 10 15:42:03.017140 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.017081 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 10 15:42:03.023590 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.023483 2576 manager.go:216] Machine: {Timestamp:2026-09-10 15:42:03.022080927 +0000 UTC m=+0.375600080 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3060928 MemoryCapacity:32812163072 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c6945d159d21e9a6b14a904c26bac SystemUUID:ec2c6945-d159-d21e-9a6b-14a904c26bac BootID:c25d3866-6d65-45cd-9453-2cac0e284667 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:32:72:29:5d:cf Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:32:72:29:5d:cf Speed:0 Mtu:9001} {Name:ovs-system MacAddress:0a:a0:c2:21:ae:33 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812163072 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 10 15:42:03.023590 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.023578 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 10 15:42:03.023719 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.023658 2576 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 10 15:42:03.025568 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.025550 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 10 15:42:03.027579 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.027558 2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 10 15:42:03.027707 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.027581 2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-236.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 10 15:42:03.029096 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.029085 2576 topology_manager.go:138] "Creating topology manager with none policy" Sep 10 15:42:03.029131 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.029099 2576 container_manager_linux.go:309] "Creating device plugin manager" Sep 10 15:42:03.029131 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.029110 2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 10 15:42:03.029971 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.029961 2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 10 15:42:03.031844 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.031835 2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 10 15:42:03.036640 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.036628 2576 state_mem.go:36] "Initialized new in-memory state store" Sep 10 15:42:03.036800 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.036789 2576 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 10 15:42:03.036834 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.036827 2576 kubelet.go:486] "Attempting to sync node with API server" Sep 10 15:42:03.036861 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.036848 2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 10 15:42:03.036893 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.036866 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 10 15:42:03.036893 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.036879 2576 kubelet.go:398] "Adding apiserver pod source" Sep 10 15:42:03.036893 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.036890 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 10 15:42:03.037450 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.037293 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 10 15:42:03.039252 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.039233 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 10 15:42:03.042078 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.042056 2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 10 15:42:03.044995 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.044981 2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 10 15:42:03.045052 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.045012 2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 10 15:42:03.045186 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.045175 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 10 15:42:03.045218 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.045191 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 10 15:42:03.045218 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.045197 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 10 15:42:03.045218 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.045203 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 10 15:42:03.045218 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.045214 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 10 15:42:03.045339 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.045222 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 10 15:42:03.045898 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.045885 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 10 15:42:03.045933 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.045899 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 10 15:42:03.045933 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.045906 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 10 15:42:03.045933 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.045912 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 10 15:42:03.045933 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.045921 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 10 15:42:03.046036 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.045949 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 10 15:42:03.046724 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.046715 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 10 15:42:03.046724 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.046724 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 10 15:42:03.047859 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:03.047834 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 10 15:42:03.047925 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:03.047907 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-236.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 10 15:42:03.050680 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.050666 2576 server.go:1268] "Started kubelet" Sep 10 15:42:03.050838 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.050787 2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 10 15:42:03.051256 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.051211 2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 10 15:42:03.051358 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.051276 2576 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 10 15:42:03.051528 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.051511 2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-236.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 10 15:42:03.051584 ip-10-0-139-236 systemd[1]: Started Kubernetes Kubelet. Sep 10 15:42:03.052647 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.052613 2576 server.go:310] "Adding debug handlers to kubelet server" Sep 10 15:42:03.052951 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.052928 2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 10 15:42:03.055746 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:03.055700 2576 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 10 15:42:03.058404 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.058389 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 10 15:42:03.058545 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:03.057420 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-236.ec2.internal.18d3fff94418cf67 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-236.ec2.internal,UID:ip-10-0-139-236.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-236.ec2.internal,},FirstTimestamp:2026-09-10 15:42:03.050635111 +0000 UTC m=+0.404154264,LastTimestamp:2026-09-10 15:42:03.050635111 +0000 UTC m=+0.404154264,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-236.ec2.internal,}" Sep 10 15:42:03.059026 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.059010 2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 10 15:42:03.059201 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:03.059181 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-236.ec2.internal\" not found" Sep 10 15:42:03.059801 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.059783 2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 10 15:42:03.059801 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.059783 2576 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 10 15:42:03.059921 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.059808 2576 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 10 15:42:03.059921 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.059894 2576 reconstruct.go:97] "Volume reconstruction finished" Sep 10 15:42:03.059921 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.059901 2576 reconciler.go:29] "Reconciler: start to sync state" Sep 10 15:42:03.061237 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.061219 2576 factory.go:55] Registering systemd factory Sep 10 15:42:03.061326 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.061240 2576 factory.go:223] Registration of the systemd container factory successfully Sep 10 15:42:03.061590 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.061577 2576 factory.go:153] Registering CRI-O factory Sep 10 15:42:03.061590 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.061590 2576 factory.go:223] Registration of the crio container factory successfully Sep 10 15:42:03.061697 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.061639 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 10 15:42:03.061697 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.061665 2576 factory.go:103] Registering Raw factory Sep 10 15:42:03.061697 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.061679 2576 manager.go:1196] Started watching for new ooms in manager Sep 10 15:42:03.062145 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.062130 2576 manager.go:349] Starting recovery of all containers Sep 10 15:42:03.064068 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:03.064042 2576 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-236.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 10 15:42:03.064199 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:03.064173 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 10 15:42:03.067341 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.067320 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vkdjn" Sep 10 15:42:03.073905 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.073889 2576 manager.go:354] Recovery completed Sep 10 15:42:03.076942 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.076925 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vkdjn" Sep 10 15:42:03.077929 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.077916 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 15:42:03.079717 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.079701 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-236.ec2.internal" event="NodeHasSufficientMemory" Sep 10 15:42:03.079794 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.079748 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-236.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 15:42:03.079794 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.079763 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-236.ec2.internal" event="NodeHasSufficientPID" Sep 10 15:42:03.080315 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.080302 2576 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 10 15:42:03.080315 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.080310 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 15:42:03.080315 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.080315 2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 10 15:42:03.080419 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.080329 2576 state_mem.go:36] "Initialized new in-memory state store" Sep 10 15:42:03.081011 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.080995 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-236.ec2.internal" event="NodeHasSufficientMemory" Sep 10 15:42:03.081093 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.081025 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-236.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 15:42:03.081093 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.081039 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-236.ec2.internal" event="NodeHasSufficientPID" Sep 10 15:42:03.082052 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:03.081985 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-236.ec2.internal.18d3fff945d48a8d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-236.ec2.internal,UID:ip-10-0-139-236.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-236.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-236.ec2.internal,},FirstTimestamp:2026-09-10 15:42:03.079715469 +0000 UTC m=+0.433234622,LastTimestamp:2026-09-10 15:42:03.079715469 +0000 UTC m=+0.433234622,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-236.ec2.internal,}" Sep 10 15:42:03.082313 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.082302 2576 policy_none.go:49] "None policy: Start" Sep 10 15:42:03.082351 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.082317 2576 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 10 15:42:03.082351 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.082327 2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 10 15:42:03.084313 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.084302 2576 policy_none.go:47] "Start" Sep 10 15:42:03.117328 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.117312 2576 manager.go:337] "Starting Device Plugin manager" Sep 10 15:42:03.141474 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:03.117354 2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 10 15:42:03.141474 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.117366 2576 server.go:87] "Starting device plugin registration server" Sep 10 15:42:03.141474 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.117640 2576 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 10 15:42:03.141474 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.117650 2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 10 15:42:03.141474 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.117769 2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 10 15:42:03.141474 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.117858 2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 10 15:42:03.141474 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.117868 2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 10 15:42:03.141474 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:03.118303 2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 10 15:42:03.141474 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:03.118345 2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-236.ec2.internal\" not found" Sep 10 15:42:03.160970 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.160940 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 10 15:42:03.162256 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.162240 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 10 15:42:03.162312 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.162285 2576 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 10 15:42:03.162312 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.162305 2576 kubelet.go:2448] "Starting kubelet main sync loop" Sep 10 15:42:03.162387 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:03.162343 2576 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 10 15:42:03.164958 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.164941 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 15:42:03.218357 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.218272 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 15:42:03.219277 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.219260 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-236.ec2.internal" event="NodeHasSufficientMemory" Sep 10 15:42:03.219371 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.219292 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-236.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 15:42:03.219371 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.219305 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-236.ec2.internal" event="NodeHasSufficientPID" Sep 10 15:42:03.219371 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.219337 2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-236.ec2.internal" Sep 10 15:42:03.227919 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.227899 2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-236.ec2.internal" Sep 10 15:42:03.227964 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:03.227923 2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-236.ec2.internal\": node \"ip-10-0-139-236.ec2.internal\" not found" Sep 10 15:42:03.247806 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:03.247787 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-236.ec2.internal\" not found" Sep 10 15:42:03.263219 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.263193 2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-236.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-236.ec2.internal"] Sep 10 15:42:03.263273 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.263252 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 15:42:03.264012 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.263995 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-236.ec2.internal" event="NodeHasSufficientMemory" Sep 10 15:42:03.264101 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.264028 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-236.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 15:42:03.264101 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.264043 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-236.ec2.internal" event="NodeHasSufficientPID" Sep 10 15:42:03.266396 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.266382 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 15:42:03.266519 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.266505 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-236.ec2.internal" Sep 10 15:42:03.266564 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.266535 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 15:42:03.267093 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.267078 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-236.ec2.internal" event="NodeHasSufficientMemory" Sep 10 15:42:03.267168 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.267103 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-236.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 15:42:03.267168 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.267116 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-236.ec2.internal" event="NodeHasSufficientPID" Sep 10 15:42:03.267168 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.267140 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-236.ec2.internal" event="NodeHasSufficientMemory" Sep 10 15:42:03.267168 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.267161 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-236.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 15:42:03.267168 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.267171 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-236.ec2.internal" event="NodeHasSufficientPID" Sep 10 15:42:03.270708 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.270459 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-236.ec2.internal" Sep 10 15:42:03.272247 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.270724 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 15:42:03.272775 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.272636 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-236.ec2.internal" event="NodeHasSufficientMemory" Sep 10 15:42:03.272775 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.272659 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-236.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 15:42:03.272775 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.272674 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-236.ec2.internal" event="NodeHasSufficientPID" Sep 10 15:42:03.285276 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:03.285258 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-236.ec2.internal\" not found" node="ip-10-0-139-236.ec2.internal" Sep 10 15:42:03.290631 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:03.290616 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-236.ec2.internal\" not found" node="ip-10-0-139-236.ec2.internal" Sep 10 15:42:03.348436 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:03.348417 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-236.ec2.internal\" not found" Sep 10 15:42:03.361641 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.361625 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a8043e4990f69187018df43e2af02f19-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-236.ec2.internal\" (UID: \"a8043e4990f69187018df43e2af02f19\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-236.ec2.internal" Sep 10 15:42:03.361712 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.361648 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a8043e4990f69187018df43e2af02f19-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-236.ec2.internal\" (UID: \"a8043e4990f69187018df43e2af02f19\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-236.ec2.internal" Sep 10 15:42:03.361712 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.361665 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c089abe4aa32d6992a9466be84e92f1f-config\") pod \"kube-apiserver-proxy-ip-10-0-139-236.ec2.internal\" (UID: \"c089abe4aa32d6992a9466be84e92f1f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-236.ec2.internal" Sep 10 15:42:03.448895 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:03.448868 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-236.ec2.internal\" not found" Sep 10 15:42:03.462216 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.462196 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c089abe4aa32d6992a9466be84e92f1f-config\") pod \"kube-apiserver-proxy-ip-10-0-139-236.ec2.internal\" (UID: \"c089abe4aa32d6992a9466be84e92f1f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-236.ec2.internal" Sep 10 15:42:03.462270 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.462221 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a8043e4990f69187018df43e2af02f19-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-236.ec2.internal\" (UID: \"a8043e4990f69187018df43e2af02f19\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-236.ec2.internal" Sep 10 15:42:03.462270 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.462242 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a8043e4990f69187018df43e2af02f19-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-236.ec2.internal\" (UID: \"a8043e4990f69187018df43e2af02f19\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-236.ec2.internal" Sep 10 15:42:03.462270 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.462267 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a8043e4990f69187018df43e2af02f19-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-236.ec2.internal\" (UID: \"a8043e4990f69187018df43e2af02f19\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-236.ec2.internal" Sep 10 15:42:03.462362 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.462298 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c089abe4aa32d6992a9466be84e92f1f-config\") pod \"kube-apiserver-proxy-ip-10-0-139-236.ec2.internal\" (UID: \"c089abe4aa32d6992a9466be84e92f1f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-236.ec2.internal" Sep 10 15:42:03.462362 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.462305 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a8043e4990f69187018df43e2af02f19-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-236.ec2.internal\" (UID: \"a8043e4990f69187018df43e2af02f19\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-236.ec2.internal" Sep 10 15:42:03.549705 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:03.549632 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-236.ec2.internal\" not found" Sep 10 15:42:03.588172 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.588128 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-236.ec2.internal" Sep 10 15:42:03.594034 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.594018 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-236.ec2.internal" Sep 10 15:42:03.650641 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:03.650610 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-236.ec2.internal\" not found" Sep 10 15:42:03.751230 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:03.751182 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-236.ec2.internal\" not found" Sep 10 15:42:03.851771 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:03.851694 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-236.ec2.internal\" not found" Sep 10 15:42:03.944242 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.944219 2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 15:42:03.960497 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.960482 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-236.ec2.internal" Sep 10 15:42:03.971898 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.971876 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 10 15:42:03.973301 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.973284 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-236.ec2.internal" Sep 10 15:42:03.981972 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.981956 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 10 15:42:03.989126 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.989110 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 10 15:42:03.989238 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.989221 2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 10 15:42:03.989289 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:03.989251 2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 10 15:42:04.030743 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.030694 2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 15:42:04.037860 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.037840 2576 apiserver.go:52] "Watching apiserver" Sep 10 15:42:04.042759 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.042717 2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 10 15:42:04.043684 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.043664 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-hzjqc","kube-system/konnectivity-agent-5sgqq","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d46d4","openshift-dns/node-resolver-5zp9c","openshift-image-registry/node-ca-xwmfs","openshift-multus/multus-5hkzp","openshift-multus/multus-additional-cni-plugins-72k8k","openshift-network-diagnostics/network-check-target-ptjhb","kube-system/kube-apiserver-proxy-ip-10-0-139-236.ec2.internal","openshift-cluster-node-tuning-operator/tuned-hzfnv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-236.ec2.internal","openshift-multus/network-metrics-daemon-9d4jc","openshift-network-operator/iptables-alerter-xcbq4","openshift-ovn-kubernetes/ovnkube-node-zc9b9"] Sep 10 15:42:04.046181 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.046166 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hzjqc" Sep 10 15:42:04.046251 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:04.046231 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hzjqc" podUID="1b1f777f-8ed4-408b-9721-80c9ae47af6e" Sep 10 15:42:04.048301 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.048287 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5sgqq" Sep 10 15:42:04.050214 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.050194 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-9mvsp\"" Sep 10 15:42:04.050307 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.050225 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 10 15:42:04.050307 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.050249 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 10 15:42:04.050471 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.050456 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d46d4" Sep 10 15:42:04.052149 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.052135 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 10 15:42:04.052226 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.052212 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 10 15:42:04.052285 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.052232 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 10 15:42:04.052285 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.052216 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-b6cbt\"" Sep 10 15:42:04.052606 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.052592 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5zp9c" Sep 10 15:42:04.054355 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.054339 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 10 15:42:04.054440 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.054353 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bg4rj\"" Sep 10 15:42:04.054440 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.054357 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 10 15:42:04.054777 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.054764 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xwmfs" Sep 10 15:42:04.056514 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.056497 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 10 15:42:04.056611 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.056558 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 10 15:42:04.056759 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.056746 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mzpl6\"" Sep 10 15:42:04.056806 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.056751 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 10 15:42:04.056910 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.056898 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5hkzp" Sep 10 15:42:04.058520 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.058504 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 10 15:42:04.058637 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.058600 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 10 15:42:04.058710 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.058673 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 10 15:42:04.058908 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.058894 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 10 15:42:04.058957 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.058930 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-n7hmt\"" Sep 10 15:42:04.059003 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.058986 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 10 15:42:04.059147 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.059123 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-72k8k" Sep 10 15:42:04.061286 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.061270 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ptjhb" Sep 10 15:42:04.061362 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:04.061337 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ptjhb" podUID="c27ce7b1-1b71-4f92-8897-fd4748ebd262" Sep 10 15:42:04.061803 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.061783 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 10 15:42:04.061891 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.061829 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7cmh2\"" Sep 10 15:42:04.061891 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.061783 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 10 15:42:04.064317 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.064300 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hzfnv" Sep 10 15:42:04.065952 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.065932 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk7nx\" (UniqueName: \"kubernetes.io/projected/50da34c6-e20d-41cf-8f08-95c7143e087c-kube-api-access-hk7nx\") pod \"aws-ebs-csi-driver-node-d46d4\" (UID: \"50da34c6-e20d-41cf-8f08-95c7143e087c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d46d4" Sep 10 15:42:04.066054 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.065960 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b8c9295-101d-4700-a819-ae5172b0deee-tmp-dir\") pod \"node-resolver-5zp9c\" (UID: \"3b8c9295-101d-4700-a819-ae5172b0deee\") " pod="openshift-dns/node-resolver-5zp9c" Sep 10 15:42:04.066054 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.065975 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/849ac9ca-6d6b-4759-8871-2afe71e4afe3-hostroot\") pod \"multus-5hkzp\" (UID: \"849ac9ca-6d6b-4759-8871-2afe71e4afe3\") " pod="openshift-multus/multus-5hkzp" Sep 10 15:42:04.066054 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.065988 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/849ac9ca-6d6b-4759-8871-2afe71e4afe3-etc-kubernetes\") pod \"multus-5hkzp\" (UID: \"849ac9ca-6d6b-4759-8871-2afe71e4afe3\") " pod="openshift-multus/multus-5hkzp" Sep 10 15:42:04.066054 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.066004 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/50da34c6-e20d-41cf-8f08-95c7143e087c-registration-dir\") pod \"aws-ebs-csi-driver-node-d46d4\" (UID: \"50da34c6-e20d-41cf-8f08-95c7143e087c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d46d4" Sep 10 15:42:04.066054 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.066025 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/849ac9ca-6d6b-4759-8871-2afe71e4afe3-cnibin\") pod \"multus-5hkzp\" (UID: \"849ac9ca-6d6b-4759-8871-2afe71e4afe3\") " pod="openshift-multus/multus-5hkzp" Sep 10 15:42:04.066322 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.066065 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/849ac9ca-6d6b-4759-8871-2afe71e4afe3-host-var-lib-kubelet\") pod \"multus-5hkzp\" (UID: \"849ac9ca-6d6b-4759-8871-2afe71e4afe3\") " pod="openshift-multus/multus-5hkzp" Sep 10 15:42:04.066322 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.066097 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d17f9024-a71a-480f-acab-1f6e6279f055-tuning-conf-dir\") pod \"multus-additional-cni-plugins-72k8k\" (UID: \"d17f9024-a71a-480f-acab-1f6e6279f055\") " pod="openshift-multus/multus-additional-cni-plugins-72k8k" Sep 10 15:42:04.066322 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.066160 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/50da34c6-e20d-41cf-8f08-95c7143e087c-etc-selinux\") pod \"aws-ebs-csi-driver-node-d46d4\" (UID: \"50da34c6-e20d-41cf-8f08-95c7143e087c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d46d4" Sep 10 15:42:04.066322 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.066195 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9d4jc" Sep 10 15:42:04.066322 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.066194 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/50da34c6-e20d-41cf-8f08-95c7143e087c-sys-fs\") pod \"aws-ebs-csi-driver-node-d46d4\" (UID: \"50da34c6-e20d-41cf-8f08-95c7143e087c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d46d4" Sep 10 15:42:04.066322 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.066232 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/322fb473-4c72-4194-9bba-9728743cdd74-host\") pod \"node-ca-xwmfs\" (UID: \"322fb473-4c72-4194-9bba-9728743cdd74\") " pod="openshift-image-registry/node-ca-xwmfs" Sep 10 15:42:04.066322 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:04.066246 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9d4jc" podUID="bbfbb1ad-2b61-4739-926f-01774a102c98" Sep 10 15:42:04.066322 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.066256 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7cfm\" (UniqueName: \"kubernetes.io/projected/322fb473-4c72-4194-9bba-9728743cdd74-kube-api-access-x7cfm\") pod \"node-ca-xwmfs\" (UID: \"322fb473-4c72-4194-9bba-9728743cdd74\") " pod="openshift-image-registry/node-ca-xwmfs" Sep 10 15:42:04.066322 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.066291 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/849ac9ca-6d6b-4759-8871-2afe71e4afe3-host-run-netns\") pod \"multus-5hkzp\" (UID: \"849ac9ca-6d6b-4759-8871-2afe71e4afe3\") " pod="openshift-multus/multus-5hkzp" Sep 10 15:42:04.066322 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.066312 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/322fb473-4c72-4194-9bba-9728743cdd74-serviceca\") pod \"node-ca-xwmfs\" (UID: \"322fb473-4c72-4194-9bba-9728743cdd74\") " pod="openshift-image-registry/node-ca-xwmfs" Sep 10 15:42:04.066777 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.066353 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stt2n\" (UniqueName: \"kubernetes.io/projected/849ac9ca-6d6b-4759-8871-2afe71e4afe3-kube-api-access-stt2n\") pod \"multus-5hkzp\" (UID: \"849ac9ca-6d6b-4759-8871-2afe71e4afe3\") " pod="openshift-multus/multus-5hkzp" Sep 10 15:42:04.066777 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.066404 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zxhxk\"" Sep 10 15:42:04.066777 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.066408 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppv8k\" (UniqueName: \"kubernetes.io/projected/c27ce7b1-1b71-4f92-8897-fd4748ebd262-kube-api-access-ppv8k\") pod \"network-check-target-ptjhb\" (UID: \"c27ce7b1-1b71-4f92-8897-fd4748ebd262\") " pod="openshift-network-diagnostics/network-check-target-ptjhb" Sep 10 15:42:04.066777 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.066434 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1b1f777f-8ed4-408b-9721-80c9ae47af6e-kubelet-config\") pod \"global-pull-secret-syncer-hzjqc\" (UID: \"1b1f777f-8ed4-408b-9721-80c9ae47af6e\") " pod="kube-system/global-pull-secret-syncer-hzjqc" Sep 10 15:42:04.066777 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.066455 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1b1f777f-8ed4-408b-9721-80c9ae47af6e-dbus\") pod \"global-pull-secret-syncer-hzjqc\" (UID: \"1b1f777f-8ed4-408b-9721-80c9ae47af6e\") " pod="kube-system/global-pull-secret-syncer-hzjqc" Sep 10 15:42:04.066777 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.066477 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/849ac9ca-6d6b-4759-8871-2afe71e4afe3-system-cni-dir\") pod \"multus-5hkzp\" (UID: \"849ac9ca-6d6b-4759-8871-2afe71e4afe3\") " pod="openshift-multus/multus-5hkzp" Sep 10 15:42:04.066777 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.066498 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/849ac9ca-6d6b-4759-8871-2afe71e4afe3-cni-binary-copy\") pod \"multus-5hkzp\" (UID: \"849ac9ca-6d6b-4759-8871-2afe71e4afe3\") " pod="openshift-multus/multus-5hkzp" Sep 10 15:42:04.066777 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.066541 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d17f9024-a71a-480f-acab-1f6e6279f055-cni-binary-copy\") pod \"multus-additional-cni-plugins-72k8k\" (UID: \"d17f9024-a71a-480f-acab-1f6e6279f055\") " pod="openshift-multus/multus-additional-cni-plugins-72k8k" Sep 10 15:42:04.066777 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.066561 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d17f9024-a71a-480f-acab-1f6e6279f055-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-72k8k\" (UID: \"d17f9024-a71a-480f-acab-1f6e6279f055\") " pod="openshift-multus/multus-additional-cni-plugins-72k8k" Sep 10 15:42:04.066777 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.066592 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d17f9024-a71a-480f-acab-1f6e6279f055-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-72k8k\" (UID: \"d17f9024-a71a-480f-acab-1f6e6279f055\") " pod="openshift-multus/multus-additional-cni-plugins-72k8k" Sep 10 15:42:04.066777 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.066608 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2dlp\" (UniqueName: \"kubernetes.io/projected/d17f9024-a71a-480f-acab-1f6e6279f055-kube-api-access-d2dlp\") pod \"multus-additional-cni-plugins-72k8k\" (UID: \"d17f9024-a71a-480f-acab-1f6e6279f055\") " pod="openshift-multus/multus-additional-cni-plugins-72k8k" Sep 10 15:42:04.066777 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.066624 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a-konnectivity-ca\") pod \"konnectivity-agent-5sgqq\" (UID: \"a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a\") " pod="kube-system/konnectivity-agent-5sgqq" Sep 10 15:42:04.066777 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.066637 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3b8c9295-101d-4700-a819-ae5172b0deee-hosts-file\") pod \"node-resolver-5zp9c\" (UID: \"3b8c9295-101d-4700-a819-ae5172b0deee\") " pod="openshift-dns/node-resolver-5zp9c" Sep 10 15:42:04.066777 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.066681 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/849ac9ca-6d6b-4759-8871-2afe71e4afe3-host-var-lib-cni-multus\") pod \"multus-5hkzp\" (UID: \"849ac9ca-6d6b-4759-8871-2afe71e4afe3\") " pod="openshift-multus/multus-5hkzp" Sep 10 15:42:04.066777 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.066718 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d17f9024-a71a-480f-acab-1f6e6279f055-system-cni-dir\") pod \"multus-additional-cni-plugins-72k8k\" (UID: \"d17f9024-a71a-480f-acab-1f6e6279f055\") " pod="openshift-multus/multus-additional-cni-plugins-72k8k" Sep 10 15:42:04.066777 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.066776 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d17f9024-a71a-480f-acab-1f6e6279f055-cnibin\") pod \"multus-additional-cni-plugins-72k8k\" (UID: \"d17f9024-a71a-480f-acab-1f6e6279f055\") " pod="openshift-multus/multus-additional-cni-plugins-72k8k" Sep 10 15:42:04.067411 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.066807 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/50da34c6-e20d-41cf-8f08-95c7143e087c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d46d4\" (UID: \"50da34c6-e20d-41cf-8f08-95c7143e087c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d46d4" Sep 10 15:42:04.067411 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.066814 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 10 15:42:04.067411 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.066826 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/50da34c6-e20d-41cf-8f08-95c7143e087c-socket-dir\") pod \"aws-ebs-csi-driver-node-d46d4\" (UID: \"50da34c6-e20d-41cf-8f08-95c7143e087c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d46d4" Sep 10 15:42:04.067411 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.066844 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a-agent-certs\") pod \"konnectivity-agent-5sgqq\" (UID: \"a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a\") " pod="kube-system/konnectivity-agent-5sgqq" Sep 10 15:42:04.067411 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.066851 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 10 15:42:04.067411 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.066860 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/849ac9ca-6d6b-4759-8871-2afe71e4afe3-multus-socket-dir-parent\") pod \"multus-5hkzp\" (UID: \"849ac9ca-6d6b-4759-8871-2afe71e4afe3\") " pod="openshift-multus/multus-5hkzp" Sep 10 15:42:04.067411 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.066908 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/849ac9ca-6d6b-4759-8871-2afe71e4afe3-host-var-lib-cni-bin\") pod \"multus-5hkzp\" (UID: \"849ac9ca-6d6b-4759-8871-2afe71e4afe3\") " pod="openshift-multus/multus-5hkzp" Sep 10 15:42:04.067411 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.066936 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/849ac9ca-6d6b-4759-8871-2afe71e4afe3-multus-daemon-config\") pod \"multus-5hkzp\" (UID: \"849ac9ca-6d6b-4759-8871-2afe71e4afe3\") " pod="openshift-multus/multus-5hkzp" Sep 10 15:42:04.067411 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.066958 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1b1f777f-8ed4-408b-9721-80c9ae47af6e-original-pull-secret\") pod \"global-pull-secret-syncer-hzjqc\" (UID: \"1b1f777f-8ed4-408b-9721-80c9ae47af6e\") " pod="kube-system/global-pull-secret-syncer-hzjqc" Sep 10 15:42:04.067411 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.066982 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/50da34c6-e20d-41cf-8f08-95c7143e087c-device-dir\") pod \"aws-ebs-csi-driver-node-d46d4\" (UID: \"50da34c6-e20d-41cf-8f08-95c7143e087c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d46d4" Sep 10 15:42:04.067411 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.067004 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvm9n\" (UniqueName: \"kubernetes.io/projected/3b8c9295-101d-4700-a819-ae5172b0deee-kube-api-access-dvm9n\") pod \"node-resolver-5zp9c\" (UID: \"3b8c9295-101d-4700-a819-ae5172b0deee\") " pod="openshift-dns/node-resolver-5zp9c" Sep 10 15:42:04.067411 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.067026 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/849ac9ca-6d6b-4759-8871-2afe71e4afe3-multus-cni-dir\") pod \"multus-5hkzp\" (UID: \"849ac9ca-6d6b-4759-8871-2afe71e4afe3\") " pod="openshift-multus/multus-5hkzp" Sep 10 15:42:04.067411 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.067040 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/849ac9ca-6d6b-4759-8871-2afe71e4afe3-host-run-k8s-cni-cncf-io\") pod \"multus-5hkzp\" (UID: \"849ac9ca-6d6b-4759-8871-2afe71e4afe3\") " pod="openshift-multus/multus-5hkzp" Sep 10 15:42:04.067411 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.067060 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/849ac9ca-6d6b-4759-8871-2afe71e4afe3-multus-conf-dir\") pod \"multus-5hkzp\" (UID: \"849ac9ca-6d6b-4759-8871-2afe71e4afe3\") " pod="openshift-multus/multus-5hkzp" Sep 10 15:42:04.067411 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.067123 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/849ac9ca-6d6b-4759-8871-2afe71e4afe3-host-run-multus-certs\") pod \"multus-5hkzp\" (UID: \"849ac9ca-6d6b-4759-8871-2afe71e4afe3\") " pod="openshift-multus/multus-5hkzp" Sep 10 15:42:04.068347 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.068227 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xcbq4" Sep 10 15:42:04.069560 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.069543 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 10 15:42:04.070051 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.070036 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 10 15:42:04.070133 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.070120 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 10 15:42:04.070172 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.070123 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 10 15:42:04.070461 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.070446 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4d5k2\"" Sep 10 15:42:04.070493 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.070473 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:04.072308 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.072291 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 10 15:42:04.072556 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.072537 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mww2g\"" Sep 10 15:42:04.072646 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.072556 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 10 15:42:04.072646 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.072606 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 10 15:42:04.072761 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.072685 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 10 15:42:04.072821 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.072762 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 10 15:42:04.072866 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.072840 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 10 15:42:04.080075 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.080039 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 15:37:03 +0000 UTC" deadline="2028-05-13 19:17:45.483819667 +0000 UTC" Sep 10 15:42:04.080075 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.080073 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14667h35m41.403748569s" Sep 10 15:42:04.094647 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.094623 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-6vzvh" Sep 10 15:42:04.099577 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.099557 2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 15:42:04.103352 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.103290 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-6vzvh" Sep 10 15:42:04.161018 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.160994 2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 10 15:42:04.168169 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.168143 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dd44752e-23f1-4ebe-a1ce-f7ad0561fc59-host-slash\") pod \"ovnkube-node-zc9b9\" (UID: \"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59\") " pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:04.168294 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.168177 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7cfm\" (UniqueName: \"kubernetes.io/projected/322fb473-4c72-4194-9bba-9728743cdd74-kube-api-access-x7cfm\") pod \"node-ca-xwmfs\" (UID: \"322fb473-4c72-4194-9bba-9728743cdd74\") " pod="openshift-image-registry/node-ca-xwmfs" Sep 10 15:42:04.168294 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.168192 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb-etc-sysctl-d\") pod \"tuned-hzfnv\" (UID: \"2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-hzfnv" Sep 10 15:42:04.168294 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.168205 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dd44752e-23f1-4ebe-a1ce-f7ad0561fc59-host-run-netns\") pod \"ovnkube-node-zc9b9\" (UID: \"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59\") " pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:04.168294 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.168219 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd44752e-23f1-4ebe-a1ce-f7ad0561fc59-run-openvswitch\") pod \"ovnkube-node-zc9b9\" (UID: \"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59\") " pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:04.168496 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.168323 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dd44752e-23f1-4ebe-a1ce-f7ad0561fc59-host-cni-bin\") pod \"ovnkube-node-zc9b9\" (UID: \"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59\") " pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:04.168496 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.168353 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dd44752e-23f1-4ebe-a1ce-f7ad0561fc59-env-overrides\") pod \"ovnkube-node-zc9b9\" (UID: \"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59\") " pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:04.168496 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.168386 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/322fb473-4c72-4194-9bba-9728743cdd74-serviceca\") pod \"node-ca-xwmfs\" (UID: \"322fb473-4c72-4194-9bba-9728743cdd74\") " pod="openshift-image-registry/node-ca-xwmfs" Sep 10 15:42:04.168496 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.168434 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0fb1831f-d8df-47b6-aeb9-21565c4ffe9e-iptables-alerter-script\") pod \"iptables-alerter-xcbq4\" (UID: \"0fb1831f-d8df-47b6-aeb9-21565c4ffe9e\") " pod="openshift-network-operator/iptables-alerter-xcbq4" Sep 10 15:42:04.168496 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.168475 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxq2x\" (UniqueName: \"kubernetes.io/projected/0fb1831f-d8df-47b6-aeb9-21565c4ffe9e-kube-api-access-wxq2x\") pod \"iptables-alerter-xcbq4\" (UID: \"0fb1831f-d8df-47b6-aeb9-21565c4ffe9e\") " pod="openshift-network-operator/iptables-alerter-xcbq4" Sep 10 15:42:04.168697 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.168500 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd44752e-23f1-4ebe-a1ce-f7ad0561fc59-etc-openvswitch\") pod \"ovnkube-node-zc9b9\" (UID: \"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59\") " pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:04.168697 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.168523 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dd44752e-23f1-4ebe-a1ce-f7ad0561fc59-ovnkube-config\") pod \"ovnkube-node-zc9b9\" (UID: \"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59\") " pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:04.168697 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.168552 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stt2n\" (UniqueName: \"kubernetes.io/projected/849ac9ca-6d6b-4759-8871-2afe71e4afe3-kube-api-access-stt2n\") pod \"multus-5hkzp\" (UID: \"849ac9ca-6d6b-4759-8871-2afe71e4afe3\") " pod="openshift-multus/multus-5hkzp" Sep 10 15:42:04.168697 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.168584 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1b1f777f-8ed4-408b-9721-80c9ae47af6e-kubelet-config\") pod \"global-pull-secret-syncer-hzjqc\" (UID: \"1b1f777f-8ed4-408b-9721-80c9ae47af6e\") " pod="kube-system/global-pull-secret-syncer-hzjqc" Sep 10 15:42:04.168697 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.168610 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1b1f777f-8ed4-408b-9721-80c9ae47af6e-dbus\") pod \"global-pull-secret-syncer-hzjqc\" (UID: \"1b1f777f-8ed4-408b-9721-80c9ae47af6e\") " pod="kube-system/global-pull-secret-syncer-hzjqc" Sep 10 15:42:04.168697 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.168637 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb-etc-sysctl-conf\") pod \"tuned-hzfnv\" (UID: \"2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-hzfnv" Sep 10 15:42:04.168697 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.168665 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb-tmp\") pod \"tuned-hzfnv\" (UID: \"2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-hzfnv" Sep 10 15:42:04.168697 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.168686 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1b1f777f-8ed4-408b-9721-80c9ae47af6e-kubelet-config\") pod \"global-pull-secret-syncer-hzjqc\" (UID: \"1b1f777f-8ed4-408b-9721-80c9ae47af6e\") " pod="kube-system/global-pull-secret-syncer-hzjqc" Sep 10 15:42:04.168697 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.168694 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1b1f777f-8ed4-408b-9721-80c9ae47af6e-dbus\") pod \"global-pull-secret-syncer-hzjqc\" (UID: \"1b1f777f-8ed4-408b-9721-80c9ae47af6e\") " pod="kube-system/global-pull-secret-syncer-hzjqc" Sep 10 15:42:04.169139 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.168690 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/849ac9ca-6d6b-4759-8871-2afe71e4afe3-system-cni-dir\") pod \"multus-5hkzp\" (UID: \"849ac9ca-6d6b-4759-8871-2afe71e4afe3\") " pod="openshift-multus/multus-5hkzp" Sep 10 15:42:04.169139 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.168752 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/849ac9ca-6d6b-4759-8871-2afe71e4afe3-cni-binary-copy\") pod \"multus-5hkzp\" (UID: \"849ac9ca-6d6b-4759-8871-2afe71e4afe3\") " pod="openshift-multus/multus-5hkzp" Sep 10 15:42:04.169139 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.168788 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d17f9024-a71a-480f-acab-1f6e6279f055-cni-binary-copy\") pod \"multus-additional-cni-plugins-72k8k\" (UID: \"d17f9024-a71a-480f-acab-1f6e6279f055\") " pod="openshift-multus/multus-additional-cni-plugins-72k8k" Sep 10 15:42:04.169139 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.168810 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/849ac9ca-6d6b-4759-8871-2afe71e4afe3-system-cni-dir\") pod \"multus-5hkzp\" (UID: \"849ac9ca-6d6b-4759-8871-2afe71e4afe3\") " pod="openshift-multus/multus-5hkzp" Sep 10 15:42:04.169139 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.168816 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d17f9024-a71a-480f-acab-1f6e6279f055-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-72k8k\" (UID: \"d17f9024-a71a-480f-acab-1f6e6279f055\") " pod="openshift-multus/multus-additional-cni-plugins-72k8k" Sep 10 15:42:04.169139 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.168838 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/322fb473-4c72-4194-9bba-9728743cdd74-serviceca\") pod \"node-ca-xwmfs\" (UID: \"322fb473-4c72-4194-9bba-9728743cdd74\") " pod="openshift-image-registry/node-ca-xwmfs" Sep 10 15:42:04.169139 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.168854 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb-etc-modprobe-d\") pod \"tuned-hzfnv\" (UID: \"2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-hzfnv" Sep 10 15:42:04.169139 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.168885 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dd44752e-23f1-4ebe-a1ce-f7ad0561fc59-run-systemd\") pod \"ovnkube-node-zc9b9\" (UID: \"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59\") " pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:04.169139 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.168934 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd44752e-23f1-4ebe-a1ce-f7ad0561fc59-host-run-ovn-kubernetes\") pod \"ovnkube-node-zc9b9\" (UID: \"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59\") " pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:04.169139 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.168969 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dd44752e-23f1-4ebe-a1ce-f7ad0561fc59-ovn-node-metrics-cert\") pod \"ovnkube-node-zc9b9\" (UID: \"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59\") " pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:04.169139 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.168999 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a-konnectivity-ca\") pod \"konnectivity-agent-5sgqq\" (UID: \"a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a\") " pod="kube-system/konnectivity-agent-5sgqq" Sep 10 15:42:04.169139 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.169023 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3b8c9295-101d-4700-a819-ae5172b0deee-hosts-file\") pod \"node-resolver-5zp9c\" (UID: \"3b8c9295-101d-4700-a819-ae5172b0deee\") " pod="openshift-dns/node-resolver-5zp9c" Sep 10 15:42:04.169139 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.169046 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/849ac9ca-6d6b-4759-8871-2afe71e4afe3-host-var-lib-cni-multus\") pod \"multus-5hkzp\" (UID: \"849ac9ca-6d6b-4759-8871-2afe71e4afe3\") " pod="openshift-multus/multus-5hkzp" Sep 10 15:42:04.169139 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.169088 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d17f9024-a71a-480f-acab-1f6e6279f055-system-cni-dir\") pod \"multus-additional-cni-plugins-72k8k\" (UID: \"d17f9024-a71a-480f-acab-1f6e6279f055\") " pod="openshift-multus/multus-additional-cni-plugins-72k8k" Sep 10 15:42:04.169139 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.169141 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb-etc-sysconfig\") pod \"tuned-hzfnv\" (UID: \"2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-hzfnv" Sep 10 15:42:04.169886 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.169144 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3b8c9295-101d-4700-a819-ae5172b0deee-hosts-file\") pod \"node-resolver-5zp9c\" (UID: \"3b8c9295-101d-4700-a819-ae5172b0deee\") " pod="openshift-dns/node-resolver-5zp9c" Sep 10 15:42:04.169886 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.169163 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb-host\") pod \"tuned-hzfnv\" (UID: \"2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-hzfnv" Sep 10 15:42:04.169886 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.169186 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dd44752e-23f1-4ebe-a1ce-f7ad0561fc59-log-socket\") pod \"ovnkube-node-zc9b9\" (UID: \"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59\") " pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:04.169886 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.169210 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a-agent-certs\") pod \"konnectivity-agent-5sgqq\" (UID: \"a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a\") " pod="kube-system/konnectivity-agent-5sgqq" Sep 10 15:42:04.169886 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.169235 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/849ac9ca-6d6b-4759-8871-2afe71e4afe3-multus-daemon-config\") pod \"multus-5hkzp\" (UID: \"849ac9ca-6d6b-4759-8871-2afe71e4afe3\") " pod="openshift-multus/multus-5hkzp" Sep 10 15:42:04.169886 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.169290 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/849ac9ca-6d6b-4759-8871-2afe71e4afe3-cni-binary-copy\") pod \"multus-5hkzp\" (UID: \"849ac9ca-6d6b-4759-8871-2afe71e4afe3\") " pod="openshift-multus/multus-5hkzp" Sep 10 15:42:04.169886 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.169207 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/849ac9ca-6d6b-4759-8871-2afe71e4afe3-host-var-lib-cni-multus\") pod \"multus-5hkzp\" (UID: \"849ac9ca-6d6b-4759-8871-2afe71e4afe3\") " pod="openshift-multus/multus-5hkzp" Sep 10 15:42:04.169886 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.169261 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb-var-lib-kubelet\") pod \"tuned-hzfnv\" (UID: \"2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-hzfnv" Sep 10 15:42:04.169886 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.169346 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbvhl\" (UniqueName: \"kubernetes.io/projected/bbfbb1ad-2b61-4739-926f-01774a102c98-kube-api-access-kbvhl\") pod \"network-metrics-daemon-9d4jc\" (UID: \"bbfbb1ad-2b61-4739-926f-01774a102c98\") " pod="openshift-multus/network-metrics-daemon-9d4jc" Sep 10 15:42:04.169886 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.169373 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dd44752e-23f1-4ebe-a1ce-f7ad0561fc59-systemd-units\") pod \"ovnkube-node-zc9b9\" (UID: \"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59\") " pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:04.169886 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.169410 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/849ac9ca-6d6b-4759-8871-2afe71e4afe3-multus-cni-dir\") pod \"multus-5hkzp\" (UID: \"849ac9ca-6d6b-4759-8871-2afe71e4afe3\") " pod="openshift-multus/multus-5hkzp" Sep 10 15:42:04.169886 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.169439 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/849ac9ca-6d6b-4759-8871-2afe71e4afe3-host-run-k8s-cni-cncf-io\") pod \"multus-5hkzp\" (UID: \"849ac9ca-6d6b-4759-8871-2afe71e4afe3\") " pod="openshift-multus/multus-5hkzp" Sep 10 15:42:04.169886 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.169442 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d17f9024-a71a-480f-acab-1f6e6279f055-cni-binary-copy\") pod \"multus-additional-cni-plugins-72k8k\" (UID: \"d17f9024-a71a-480f-acab-1f6e6279f055\") " pod="openshift-multus/multus-additional-cni-plugins-72k8k" Sep 10 15:42:04.169886 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.169452 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d17f9024-a71a-480f-acab-1f6e6279f055-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-72k8k\" (UID: \"d17f9024-a71a-480f-acab-1f6e6279f055\") " pod="openshift-multus/multus-additional-cni-plugins-72k8k" Sep 10 15:42:04.169886 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.169463 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk7nx\" (UniqueName: \"kubernetes.io/projected/50da34c6-e20d-41cf-8f08-95c7143e087c-kube-api-access-hk7nx\") pod \"aws-ebs-csi-driver-node-d46d4\" (UID: \"50da34c6-e20d-41cf-8f08-95c7143e087c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d46d4" Sep 10 15:42:04.169886 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.169513 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb-etc-kubernetes\") pod \"tuned-hzfnv\" (UID: \"2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-hzfnv" Sep 10 15:42:04.169886 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.169521 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/849ac9ca-6d6b-4759-8871-2afe71e4afe3-host-run-k8s-cni-cncf-io\") pod \"multus-5hkzp\" (UID: \"849ac9ca-6d6b-4759-8871-2afe71e4afe3\") " pod="openshift-multus/multus-5hkzp" Sep 10 15:42:04.170641 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.169541 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a-konnectivity-ca\") pod \"konnectivity-agent-5sgqq\" (UID: \"a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a\") " pod="kube-system/konnectivity-agent-5sgqq" Sep 10 15:42:04.170641 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.169546 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0fb1831f-d8df-47b6-aeb9-21565c4ffe9e-host-slash\") pod \"iptables-alerter-xcbq4\" (UID: \"0fb1831f-d8df-47b6-aeb9-21565c4ffe9e\") " pod="openshift-network-operator/iptables-alerter-xcbq4" Sep 10 15:42:04.170641 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.169567 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d17f9024-a71a-480f-acab-1f6e6279f055-system-cni-dir\") pod \"multus-additional-cni-plugins-72k8k\" (UID: \"d17f9024-a71a-480f-acab-1f6e6279f055\") " pod="openshift-multus/multus-additional-cni-plugins-72k8k" Sep 10 15:42:04.170641 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.169578 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dd44752e-23f1-4ebe-a1ce-f7ad0561fc59-node-log\") pod \"ovnkube-node-zc9b9\" (UID: \"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59\") " pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:04.170641 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.169609 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b8c9295-101d-4700-a819-ae5172b0deee-tmp-dir\") pod \"node-resolver-5zp9c\" (UID: \"3b8c9295-101d-4700-a819-ae5172b0deee\") " pod="openshift-dns/node-resolver-5zp9c" Sep 10 15:42:04.170641 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.169618 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/849ac9ca-6d6b-4759-8871-2afe71e4afe3-multus-cni-dir\") pod \"multus-5hkzp\" (UID: \"849ac9ca-6d6b-4759-8871-2afe71e4afe3\") " pod="openshift-multus/multus-5hkzp" Sep 10 15:42:04.170641 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.169624 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/849ac9ca-6d6b-4759-8871-2afe71e4afe3-hostroot\") pod \"multus-5hkzp\" (UID: \"849ac9ca-6d6b-4759-8871-2afe71e4afe3\") " pod="openshift-multus/multus-5hkzp" Sep 10 15:42:04.170641 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.169654 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/849ac9ca-6d6b-4759-8871-2afe71e4afe3-hostroot\") pod \"multus-5hkzp\" (UID: \"849ac9ca-6d6b-4759-8871-2afe71e4afe3\") " pod="openshift-multus/multus-5hkzp" Sep 10 15:42:04.170641 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.169654 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dd44752e-23f1-4ebe-a1ce-f7ad0561fc59-host-cni-netd\") pod \"ovnkube-node-zc9b9\" (UID: \"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59\") " pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:04.170641 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.169674 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/849ac9ca-6d6b-4759-8871-2afe71e4afe3-cnibin\") pod \"multus-5hkzp\" (UID: \"849ac9ca-6d6b-4759-8871-2afe71e4afe3\") " pod="openshift-multus/multus-5hkzp" Sep 10 15:42:04.170641 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.169684 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/849ac9ca-6d6b-4759-8871-2afe71e4afe3-multus-daemon-config\") pod \"multus-5hkzp\" (UID: \"849ac9ca-6d6b-4759-8871-2afe71e4afe3\") " pod="openshift-multus/multus-5hkzp" Sep 10 15:42:04.170641 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.169771 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/849ac9ca-6d6b-4759-8871-2afe71e4afe3-host-var-lib-kubelet\") pod \"multus-5hkzp\" (UID: \"849ac9ca-6d6b-4759-8871-2afe71e4afe3\") " pod="openshift-multus/multus-5hkzp" Sep 10 15:42:04.170641 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.169801 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/849ac9ca-6d6b-4759-8871-2afe71e4afe3-cnibin\") pod \"multus-5hkzp\" (UID: \"849ac9ca-6d6b-4759-8871-2afe71e4afe3\") " pod="openshift-multus/multus-5hkzp" Sep 10 15:42:04.170641 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.169844 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d17f9024-a71a-480f-acab-1f6e6279f055-tuning-conf-dir\") pod \"multus-additional-cni-plugins-72k8k\" (UID: \"d17f9024-a71a-480f-acab-1f6e6279f055\") " pod="openshift-multus/multus-additional-cni-plugins-72k8k" Sep 10 15:42:04.170641 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.169853 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/849ac9ca-6d6b-4759-8871-2afe71e4afe3-host-var-lib-kubelet\") pod \"multus-5hkzp\" (UID: \"849ac9ca-6d6b-4759-8871-2afe71e4afe3\") " pod="openshift-multus/multus-5hkzp" Sep 10 15:42:04.170641 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.169885 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/50da34c6-e20d-41cf-8f08-95c7143e087c-etc-selinux\") pod \"aws-ebs-csi-driver-node-d46d4\" (UID: \"50da34c6-e20d-41cf-8f08-95c7143e087c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d46d4" Sep 10 15:42:04.170641 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.169885 2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 10 15:42:04.170641 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.169926 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/50da34c6-e20d-41cf-8f08-95c7143e087c-sys-fs\") pod \"aws-ebs-csi-driver-node-d46d4\" (UID: \"50da34c6-e20d-41cf-8f08-95c7143e087c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d46d4" Sep 10 15:42:04.171360 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.169962 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/322fb473-4c72-4194-9bba-9728743cdd74-host\") pod \"node-ca-xwmfs\" (UID: \"322fb473-4c72-4194-9bba-9728743cdd74\") " pod="openshift-image-registry/node-ca-xwmfs" Sep 10 15:42:04.171360 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.169999 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/50da34c6-e20d-41cf-8f08-95c7143e087c-etc-selinux\") pod \"aws-ebs-csi-driver-node-d46d4\" (UID: \"50da34c6-e20d-41cf-8f08-95c7143e087c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d46d4" Sep 10 15:42:04.171360 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.170003 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dd44752e-23f1-4ebe-a1ce-f7ad0561fc59-run-ovn\") pod \"ovnkube-node-zc9b9\" (UID: \"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59\") " pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:04.171360 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.169981 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b8c9295-101d-4700-a819-ae5172b0deee-tmp-dir\") pod \"node-resolver-5zp9c\" (UID: \"3b8c9295-101d-4700-a819-ae5172b0deee\") " pod="openshift-dns/node-resolver-5zp9c" Sep 10 15:42:04.171360 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.170039 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/322fb473-4c72-4194-9bba-9728743cdd74-host\") pod \"node-ca-xwmfs\" (UID: \"322fb473-4c72-4194-9bba-9728743cdd74\") " pod="openshift-image-registry/node-ca-xwmfs" Sep 10 15:42:04.171360 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.170045 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/50da34c6-e20d-41cf-8f08-95c7143e087c-sys-fs\") pod \"aws-ebs-csi-driver-node-d46d4\" (UID: \"50da34c6-e20d-41cf-8f08-95c7143e087c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d46d4" Sep 10 15:42:04.171360 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.170051 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/849ac9ca-6d6b-4759-8871-2afe71e4afe3-host-run-netns\") pod \"multus-5hkzp\" (UID: \"849ac9ca-6d6b-4759-8871-2afe71e4afe3\") " pod="openshift-multus/multus-5hkzp" Sep 10 15:42:04.171360 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.170047 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d17f9024-a71a-480f-acab-1f6e6279f055-tuning-conf-dir\") pod \"multus-additional-cni-plugins-72k8k\" (UID: \"d17f9024-a71a-480f-acab-1f6e6279f055\") " pod="openshift-multus/multus-additional-cni-plugins-72k8k" Sep 10 15:42:04.171360 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.170087 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v8xd\" (UniqueName: \"kubernetes.io/projected/2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb-kube-api-access-2v8xd\") pod \"tuned-hzfnv\" (UID: \"2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-hzfnv" Sep 10 15:42:04.171360 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.170114 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/849ac9ca-6d6b-4759-8871-2afe71e4afe3-host-run-netns\") pod \"multus-5hkzp\" (UID: \"849ac9ca-6d6b-4759-8871-2afe71e4afe3\") " pod="openshift-multus/multus-5hkzp" Sep 10 15:42:04.171360 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.170147 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbfbb1ad-2b61-4739-926f-01774a102c98-metrics-certs\") pod \"network-metrics-daemon-9d4jc\" (UID: \"bbfbb1ad-2b61-4739-926f-01774a102c98\") " pod="openshift-multus/network-metrics-daemon-9d4jc" Sep 10 15:42:04.171360 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.170176 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppv8k\" (UniqueName: \"kubernetes.io/projected/c27ce7b1-1b71-4f92-8897-fd4748ebd262-kube-api-access-ppv8k\") pod \"network-check-target-ptjhb\" (UID: \"c27ce7b1-1b71-4f92-8897-fd4748ebd262\") " pod="openshift-network-diagnostics/network-check-target-ptjhb" Sep 10 15:42:04.171360 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.170197 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb-run\") pod \"tuned-hzfnv\" (UID: \"2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-hzfnv" Sep 10 15:42:04.171360 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.170218 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb-etc-tuned\") pod \"tuned-hzfnv\" (UID: \"2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-hzfnv" Sep 10 15:42:04.171360 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.170241 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd44752e-23f1-4ebe-a1ce-f7ad0561fc59-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zc9b9\" (UID: \"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59\") " pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:04.171360 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.170285 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d17f9024-a71a-480f-acab-1f6e6279f055-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-72k8k\" (UID: \"d17f9024-a71a-480f-acab-1f6e6279f055\") " pod="openshift-multus/multus-additional-cni-plugins-72k8k" Sep 10 15:42:04.171360 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.170311 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2dlp\" (UniqueName: \"kubernetes.io/projected/d17f9024-a71a-480f-acab-1f6e6279f055-kube-api-access-d2dlp\") pod \"multus-additional-cni-plugins-72k8k\" (UID: \"d17f9024-a71a-480f-acab-1f6e6279f055\") " pod="openshift-multus/multus-additional-cni-plugins-72k8k" Sep 10 15:42:04.171862 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.170339 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb-lib-modules\") pod \"tuned-hzfnv\" (UID: \"2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-hzfnv" Sep 10 15:42:04.171862 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.170444 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d17f9024-a71a-480f-acab-1f6e6279f055-cnibin\") pod \"multus-additional-cni-plugins-72k8k\" (UID: \"d17f9024-a71a-480f-acab-1f6e6279f055\") " pod="openshift-multus/multus-additional-cni-plugins-72k8k" Sep 10 15:42:04.171862 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.170468 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/50da34c6-e20d-41cf-8f08-95c7143e087c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d46d4\" (UID: \"50da34c6-e20d-41cf-8f08-95c7143e087c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d46d4" Sep 10 15:42:04.171862 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.170491 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/50da34c6-e20d-41cf-8f08-95c7143e087c-socket-dir\") pod \"aws-ebs-csi-driver-node-d46d4\" (UID: \"50da34c6-e20d-41cf-8f08-95c7143e087c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d46d4" Sep 10 15:42:04.171862 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.170508 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb-sys\") pod \"tuned-hzfnv\" (UID: \"2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-hzfnv" Sep 10 15:42:04.171862 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.170533 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd44752e-23f1-4ebe-a1ce-f7ad0561fc59-var-lib-openvswitch\") pod \"ovnkube-node-zc9b9\" (UID: \"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59\") " pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:04.171862 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.170564 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/50da34c6-e20d-41cf-8f08-95c7143e087c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d46d4\" (UID: \"50da34c6-e20d-41cf-8f08-95c7143e087c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d46d4" Sep 10 15:42:04.171862 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.170578 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dd44752e-23f1-4ebe-a1ce-f7ad0561fc59-ovnkube-script-lib\") pod \"ovnkube-node-zc9b9\" (UID: \"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59\") " pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:04.171862 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.170612 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d17f9024-a71a-480f-acab-1f6e6279f055-cnibin\") pod \"multus-additional-cni-plugins-72k8k\" (UID: \"d17f9024-a71a-480f-acab-1f6e6279f055\") " pod="openshift-multus/multus-additional-cni-plugins-72k8k" Sep 10 15:42:04.171862 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.170622 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/50da34c6-e20d-41cf-8f08-95c7143e087c-socket-dir\") pod \"aws-ebs-csi-driver-node-d46d4\" (UID: \"50da34c6-e20d-41cf-8f08-95c7143e087c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d46d4" Sep 10 15:42:04.171862 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.170604 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/849ac9ca-6d6b-4759-8871-2afe71e4afe3-multus-socket-dir-parent\") pod \"multus-5hkzp\" (UID: \"849ac9ca-6d6b-4759-8871-2afe71e4afe3\") " pod="openshift-multus/multus-5hkzp" Sep 10 15:42:04.171862 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.170676 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/849ac9ca-6d6b-4759-8871-2afe71e4afe3-host-var-lib-cni-bin\") pod \"multus-5hkzp\" (UID: \"849ac9ca-6d6b-4759-8871-2afe71e4afe3\") " pod="openshift-multus/multus-5hkzp" Sep 10 15:42:04.171862 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.170692 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1b1f777f-8ed4-408b-9721-80c9ae47af6e-original-pull-secret\") pod \"global-pull-secret-syncer-hzjqc\" (UID: \"1b1f777f-8ed4-408b-9721-80c9ae47af6e\") " pod="kube-system/global-pull-secret-syncer-hzjqc" Sep 10 15:42:04.171862 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.170758 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/50da34c6-e20d-41cf-8f08-95c7143e087c-device-dir\") pod \"aws-ebs-csi-driver-node-d46d4\" (UID: \"50da34c6-e20d-41cf-8f08-95c7143e087c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d46d4" Sep 10 15:42:04.171862 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.170770 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/849ac9ca-6d6b-4759-8871-2afe71e4afe3-multus-socket-dir-parent\") pod \"multus-5hkzp\" (UID: \"849ac9ca-6d6b-4759-8871-2afe71e4afe3\") " pod="openshift-multus/multus-5hkzp" Sep 10 15:42:04.171862 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.170784 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvm9n\" (UniqueName: \"kubernetes.io/projected/3b8c9295-101d-4700-a819-ae5172b0deee-kube-api-access-dvm9n\") pod \"node-resolver-5zp9c\" (UID: \"3b8c9295-101d-4700-a819-ae5172b0deee\") " pod="openshift-dns/node-resolver-5zp9c" Sep 10 15:42:04.171862 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.170786 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/849ac9ca-6d6b-4759-8871-2afe71e4afe3-host-var-lib-cni-bin\") pod \"multus-5hkzp\" (UID: \"849ac9ca-6d6b-4759-8871-2afe71e4afe3\") " pod="openshift-multus/multus-5hkzp" Sep 10 15:42:04.172367 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.170807 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/849ac9ca-6d6b-4759-8871-2afe71e4afe3-multus-conf-dir\") pod \"multus-5hkzp\" (UID: \"849ac9ca-6d6b-4759-8871-2afe71e4afe3\") " pod="openshift-multus/multus-5hkzp" Sep 10 15:42:04.172367 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.170817 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d17f9024-a71a-480f-acab-1f6e6279f055-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-72k8k\" (UID: \"d17f9024-a71a-480f-acab-1f6e6279f055\") " pod="openshift-multus/multus-additional-cni-plugins-72k8k" Sep 10 15:42:04.172367 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.170833 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/849ac9ca-6d6b-4759-8871-2afe71e4afe3-host-run-multus-certs\") pod \"multus-5hkzp\" (UID: \"849ac9ca-6d6b-4759-8871-2afe71e4afe3\") " pod="openshift-multus/multus-5hkzp" Sep 10 15:42:04.172367 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:04.170848 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 15:42:04.172367 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.170850 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/50da34c6-e20d-41cf-8f08-95c7143e087c-device-dir\") pod \"aws-ebs-csi-driver-node-d46d4\" (UID: \"50da34c6-e20d-41cf-8f08-95c7143e087c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d46d4" Sep 10 15:42:04.172367 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.170866 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb-etc-systemd\") pod \"tuned-hzfnv\" (UID: \"2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-hzfnv" Sep 10 15:42:04.172367 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.170870 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/849ac9ca-6d6b-4759-8871-2afe71e4afe3-host-run-multus-certs\") pod \"multus-5hkzp\" (UID: \"849ac9ca-6d6b-4759-8871-2afe71e4afe3\") " pod="openshift-multus/multus-5hkzp" Sep 10 15:42:04.172367 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.170899 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8nzf\" (UniqueName: \"kubernetes.io/projected/dd44752e-23f1-4ebe-a1ce-f7ad0561fc59-kube-api-access-j8nzf\") pod \"ovnkube-node-zc9b9\" (UID: \"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59\") " pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:04.172367 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:04.170920 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b1f777f-8ed4-408b-9721-80c9ae47af6e-original-pull-secret podName:1b1f777f-8ed4-408b-9721-80c9ae47af6e nodeName:}" failed. No retries permitted until 2026-09-10 15:42:04.670900312 +0000 UTC m=+2.024419457 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1b1f777f-8ed4-408b-9721-80c9ae47af6e-original-pull-secret") pod "global-pull-secret-syncer-hzjqc" (UID: "1b1f777f-8ed4-408b-9721-80c9ae47af6e") : object "kube-system"/"original-pull-secret" not registered Sep 10 15:42:04.172367 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.170929 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/849ac9ca-6d6b-4759-8871-2afe71e4afe3-multus-conf-dir\") pod \"multus-5hkzp\" (UID: \"849ac9ca-6d6b-4759-8871-2afe71e4afe3\") " pod="openshift-multus/multus-5hkzp" Sep 10 15:42:04.172367 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.170950 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/849ac9ca-6d6b-4759-8871-2afe71e4afe3-etc-kubernetes\") pod \"multus-5hkzp\" (UID: \"849ac9ca-6d6b-4759-8871-2afe71e4afe3\") " pod="openshift-multus/multus-5hkzp" Sep 10 15:42:04.172367 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.170973 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/50da34c6-e20d-41cf-8f08-95c7143e087c-registration-dir\") pod \"aws-ebs-csi-driver-node-d46d4\" (UID: \"50da34c6-e20d-41cf-8f08-95c7143e087c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d46d4" Sep 10 15:42:04.172367 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.170995 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd44752e-23f1-4ebe-a1ce-f7ad0561fc59-host-kubelet\") pod \"ovnkube-node-zc9b9\" (UID: \"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59\") " pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:04.172367 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.171054 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/849ac9ca-6d6b-4759-8871-2afe71e4afe3-etc-kubernetes\") pod \"multus-5hkzp\" (UID: \"849ac9ca-6d6b-4759-8871-2afe71e4afe3\") " pod="openshift-multus/multus-5hkzp" Sep 10 15:42:04.172367 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.171114 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/50da34c6-e20d-41cf-8f08-95c7143e087c-registration-dir\") pod \"aws-ebs-csi-driver-node-d46d4\" (UID: \"50da34c6-e20d-41cf-8f08-95c7143e087c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d46d4" Sep 10 15:42:04.174080 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.174061 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a-agent-certs\") pod \"konnectivity-agent-5sgqq\" (UID: \"a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a\") " pod="kube-system/konnectivity-agent-5sgqq" Sep 10 15:42:04.178075 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.178053 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7cfm\" (UniqueName: \"kubernetes.io/projected/322fb473-4c72-4194-9bba-9728743cdd74-kube-api-access-x7cfm\") pod \"node-ca-xwmfs\" (UID: \"322fb473-4c72-4194-9bba-9728743cdd74\") " pod="openshift-image-registry/node-ca-xwmfs" Sep 10 15:42:04.180129 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:04.180102 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 15:42:04.180228 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:04.180135 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 15:42:04.180228 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:04.180150 2576 projected.go:196] Error preparing data for projected volume kube-api-access-ppv8k for pod openshift-network-diagnostics/network-check-target-ptjhb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 15:42:04.180228 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:04.180215 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c27ce7b1-1b71-4f92-8897-fd4748ebd262-kube-api-access-ppv8k podName:c27ce7b1-1b71-4f92-8897-fd4748ebd262 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:04.680198081 +0000 UTC m=+2.033717223 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ppv8k" (UniqueName: "kubernetes.io/projected/c27ce7b1-1b71-4f92-8897-fd4748ebd262-kube-api-access-ppv8k") pod "network-check-target-ptjhb" (UID: "c27ce7b1-1b71-4f92-8897-fd4748ebd262") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 15:42:04.181802 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.181783 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stt2n\" (UniqueName: \"kubernetes.io/projected/849ac9ca-6d6b-4759-8871-2afe71e4afe3-kube-api-access-stt2n\") pod \"multus-5hkzp\" (UID: \"849ac9ca-6d6b-4759-8871-2afe71e4afe3\") " pod="openshift-multus/multus-5hkzp" Sep 10 15:42:04.182405 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.182272 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2dlp\" (UniqueName: \"kubernetes.io/projected/d17f9024-a71a-480f-acab-1f6e6279f055-kube-api-access-d2dlp\") pod \"multus-additional-cni-plugins-72k8k\" (UID: \"d17f9024-a71a-480f-acab-1f6e6279f055\") " pod="openshift-multus/multus-additional-cni-plugins-72k8k" Sep 10 15:42:04.182808 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.182790 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk7nx\" (UniqueName: \"kubernetes.io/projected/50da34c6-e20d-41cf-8f08-95c7143e087c-kube-api-access-hk7nx\") pod \"aws-ebs-csi-driver-node-d46d4\" (UID: \"50da34c6-e20d-41cf-8f08-95c7143e087c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d46d4" Sep 10 15:42:04.186867 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.186841 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvm9n\" (UniqueName: \"kubernetes.io/projected/3b8c9295-101d-4700-a819-ae5172b0deee-kube-api-access-dvm9n\") pod \"node-resolver-5zp9c\" (UID: \"3b8c9295-101d-4700-a819-ae5172b0deee\") " pod="openshift-dns/node-resolver-5zp9c" Sep 10 15:42:04.271423 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.271392 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dd44752e-23f1-4ebe-a1ce-f7ad0561fc59-host-slash\") pod \"ovnkube-node-zc9b9\" (UID: \"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59\") " pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:04.271536 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.271435 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb-etc-sysctl-d\") pod \"tuned-hzfnv\" (UID: \"2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-hzfnv" Sep 10 15:42:04.271536 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.271460 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dd44752e-23f1-4ebe-a1ce-f7ad0561fc59-host-run-netns\") pod \"ovnkube-node-zc9b9\" (UID: \"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59\") " pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:04.271536 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.271480 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd44752e-23f1-4ebe-a1ce-f7ad0561fc59-run-openvswitch\") pod \"ovnkube-node-zc9b9\" (UID: \"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59\") " pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:04.271536 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.271484 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dd44752e-23f1-4ebe-a1ce-f7ad0561fc59-host-slash\") pod \"ovnkube-node-zc9b9\" (UID: \"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59\") " pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:04.271536 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.271495 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dd44752e-23f1-4ebe-a1ce-f7ad0561fc59-host-cni-bin\") pod \"ovnkube-node-zc9b9\" (UID: \"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59\") " pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:04.271536 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.271536 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dd44752e-23f1-4ebe-a1ce-f7ad0561fc59-host-cni-bin\") pod \"ovnkube-node-zc9b9\" (UID: \"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59\") " pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:04.271859 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.271550 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dd44752e-23f1-4ebe-a1ce-f7ad0561fc59-env-overrides\") pod \"ovnkube-node-zc9b9\" (UID: \"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59\") " pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:04.271859 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.271565 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dd44752e-23f1-4ebe-a1ce-f7ad0561fc59-host-run-netns\") pod \"ovnkube-node-zc9b9\" (UID: \"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59\") " pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:04.271859 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.271587 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0fb1831f-d8df-47b6-aeb9-21565c4ffe9e-iptables-alerter-script\") pod \"iptables-alerter-xcbq4\" (UID: \"0fb1831f-d8df-47b6-aeb9-21565c4ffe9e\") " pod="openshift-network-operator/iptables-alerter-xcbq4" Sep 10 15:42:04.271859 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.271602 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd44752e-23f1-4ebe-a1ce-f7ad0561fc59-run-openvswitch\") pod \"ovnkube-node-zc9b9\" (UID: \"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59\") " pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:04.271859 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.271609 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxq2x\" (UniqueName: \"kubernetes.io/projected/0fb1831f-d8df-47b6-aeb9-21565c4ffe9e-kube-api-access-wxq2x\") pod \"iptables-alerter-xcbq4\" (UID: \"0fb1831f-d8df-47b6-aeb9-21565c4ffe9e\") " pod="openshift-network-operator/iptables-alerter-xcbq4" Sep 10 15:42:04.271859 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.271632 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb-etc-sysctl-d\") pod \"tuned-hzfnv\" (UID: \"2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-hzfnv" Sep 10 15:42:04.271859 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.271637 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd44752e-23f1-4ebe-a1ce-f7ad0561fc59-etc-openvswitch\") pod \"ovnkube-node-zc9b9\" (UID: \"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59\") " pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:04.271859 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.271669 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd44752e-23f1-4ebe-a1ce-f7ad0561fc59-etc-openvswitch\") pod \"ovnkube-node-zc9b9\" (UID: \"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59\") " pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:04.271859 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.271680 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dd44752e-23f1-4ebe-a1ce-f7ad0561fc59-ovnkube-config\") pod \"ovnkube-node-zc9b9\" (UID: \"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59\") " pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:04.271859 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.271710 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb-etc-sysctl-conf\") pod \"tuned-hzfnv\" (UID: \"2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-hzfnv" Sep 10 15:42:04.271859 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.271752 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb-tmp\") pod \"tuned-hzfnv\" (UID: \"2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-hzfnv" Sep 10 15:42:04.271859 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.271782 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb-etc-modprobe-d\") pod \"tuned-hzfnv\" (UID: \"2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-hzfnv" Sep 10 15:42:04.271859 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.271805 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dd44752e-23f1-4ebe-a1ce-f7ad0561fc59-run-systemd\") pod \"ovnkube-node-zc9b9\" (UID: \"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59\") " pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:04.271859 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.271831 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd44752e-23f1-4ebe-a1ce-f7ad0561fc59-host-run-ovn-kubernetes\") pod \"ovnkube-node-zc9b9\" (UID: \"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59\") " pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:04.271859 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.271844 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb-etc-sysctl-conf\") pod \"tuned-hzfnv\" (UID: \"2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-hzfnv" Sep 10 15:42:04.271859 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.271857 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dd44752e-23f1-4ebe-a1ce-f7ad0561fc59-ovn-node-metrics-cert\") pod \"ovnkube-node-zc9b9\" (UID: \"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59\") " pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:04.272582 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.271892 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dd44752e-23f1-4ebe-a1ce-f7ad0561fc59-run-systemd\") pod \"ovnkube-node-zc9b9\" (UID: \"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59\") " pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:04.272582 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.271921 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb-etc-modprobe-d\") pod \"tuned-hzfnv\" (UID: \"2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-hzfnv" Sep 10 15:42:04.272582 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.271923 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd44752e-23f1-4ebe-a1ce-f7ad0561fc59-host-run-ovn-kubernetes\") pod \"ovnkube-node-zc9b9\" (UID: \"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59\") " pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:04.272582 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.271928 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb-etc-sysconfig\") pod \"tuned-hzfnv\" (UID: \"2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-hzfnv" Sep 10 15:42:04.272582 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.271981 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb-host\") pod \"tuned-hzfnv\" (UID: \"2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-hzfnv" Sep 10 15:42:04.272582 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.272005 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dd44752e-23f1-4ebe-a1ce-f7ad0561fc59-log-socket\") pod \"ovnkube-node-zc9b9\" (UID: \"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59\") " pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:04.272582 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.271988 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb-etc-sysconfig\") pod \"tuned-hzfnv\" (UID: \"2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-hzfnv" Sep 10 15:42:04.272582 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.272033 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb-var-lib-kubelet\") pod \"tuned-hzfnv\" (UID: \"2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-hzfnv" Sep 10 15:42:04.272582 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.272058 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbvhl\" (UniqueName: \"kubernetes.io/projected/bbfbb1ad-2b61-4739-926f-01774a102c98-kube-api-access-kbvhl\") pod \"network-metrics-daemon-9d4jc\" (UID: \"bbfbb1ad-2b61-4739-926f-01774a102c98\") " pod="openshift-multus/network-metrics-daemon-9d4jc" Sep 10 15:42:04.272582 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.272067 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dd44752e-23f1-4ebe-a1ce-f7ad0561fc59-log-socket\") pod \"ovnkube-node-zc9b9\" (UID: \"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59\") " pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:04.272582 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.272077 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb-host\") pod \"tuned-hzfnv\" (UID: \"2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-hzfnv" Sep 10 15:42:04.272582 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.272077 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dd44752e-23f1-4ebe-a1ce-f7ad0561fc59-env-overrides\") pod \"ovnkube-node-zc9b9\" (UID: \"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59\") " pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:04.272582 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.272083 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dd44752e-23f1-4ebe-a1ce-f7ad0561fc59-systemd-units\") pod \"ovnkube-node-zc9b9\" (UID: \"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59\") " pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:04.272582 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.272114 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb-var-lib-kubelet\") pod \"tuned-hzfnv\" (UID: \"2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-hzfnv" Sep 10 15:42:04.272582 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.272120 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dd44752e-23f1-4ebe-a1ce-f7ad0561fc59-systemd-units\") pod \"ovnkube-node-zc9b9\" (UID: \"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59\") " pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:04.272582 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.272187 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb-etc-kubernetes\") pod \"tuned-hzfnv\" (UID: \"2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-hzfnv" Sep 10 15:42:04.272582 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.272214 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0fb1831f-d8df-47b6-aeb9-21565c4ffe9e-host-slash\") pod \"iptables-alerter-xcbq4\" (UID: \"0fb1831f-d8df-47b6-aeb9-21565c4ffe9e\") " pod="openshift-network-operator/iptables-alerter-xcbq4" Sep 10 15:42:04.272582 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.272240 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dd44752e-23f1-4ebe-a1ce-f7ad0561fc59-node-log\") pod \"ovnkube-node-zc9b9\" (UID: \"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59\") " pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:04.273453 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.272252 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb-etc-kubernetes\") pod \"tuned-hzfnv\" (UID: \"2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-hzfnv" Sep 10 15:42:04.273453 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.272262 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dd44752e-23f1-4ebe-a1ce-f7ad0561fc59-host-cni-netd\") pod \"ovnkube-node-zc9b9\" (UID: \"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59\") " pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:04.273453 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.272289 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dd44752e-23f1-4ebe-a1ce-f7ad0561fc59-run-ovn\") pod \"ovnkube-node-zc9b9\" (UID: \"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59\") " pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:04.273453 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.272315 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v8xd\" (UniqueName: \"kubernetes.io/projected/2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb-kube-api-access-2v8xd\") pod \"tuned-hzfnv\" (UID: \"2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-hzfnv" Sep 10 15:42:04.273453 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.272318 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0fb1831f-d8df-47b6-aeb9-21565c4ffe9e-host-slash\") pod \"iptables-alerter-xcbq4\" (UID: \"0fb1831f-d8df-47b6-aeb9-21565c4ffe9e\") " pod="openshift-network-operator/iptables-alerter-xcbq4" Sep 10 15:42:04.273453 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.272323 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dd44752e-23f1-4ebe-a1ce-f7ad0561fc59-host-cni-netd\") pod \"ovnkube-node-zc9b9\" (UID: \"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59\") " pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:04.273453 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.272317 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dd44752e-23f1-4ebe-a1ce-f7ad0561fc59-node-log\") pod \"ovnkube-node-zc9b9\" (UID: \"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59\") " pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:04.273453 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.272341 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbfbb1ad-2b61-4739-926f-01774a102c98-metrics-certs\") pod \"network-metrics-daemon-9d4jc\" (UID: \"bbfbb1ad-2b61-4739-926f-01774a102c98\") " pod="openshift-multus/network-metrics-daemon-9d4jc" Sep 10 15:42:04.273453 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.272340 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dd44752e-23f1-4ebe-a1ce-f7ad0561fc59-run-ovn\") pod \"ovnkube-node-zc9b9\" (UID: \"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59\") " pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:04.273453 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.272378 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb-run\") pod \"tuned-hzfnv\" (UID: \"2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-hzfnv" Sep 10 15:42:04.273453 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.272399 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb-etc-tuned\") pod \"tuned-hzfnv\" (UID: \"2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-hzfnv" Sep 10 15:42:04.273453 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.272424 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd44752e-23f1-4ebe-a1ce-f7ad0561fc59-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zc9b9\" (UID: \"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59\") " pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:04.273453 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.272445 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dd44752e-23f1-4ebe-a1ce-f7ad0561fc59-ovnkube-config\") pod \"ovnkube-node-zc9b9\" (UID: \"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59\") " pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:04.273453 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.272462 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb-lib-modules\") pod \"tuned-hzfnv\" (UID: \"2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-hzfnv" Sep 10 15:42:04.273453 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:04.272488 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 15:42:04.273453 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.272497 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb-sys\") pod \"tuned-hzfnv\" (UID: \"2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-hzfnv" Sep 10 15:42:04.273453 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.272521 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd44752e-23f1-4ebe-a1ce-f7ad0561fc59-var-lib-openvswitch\") pod \"ovnkube-node-zc9b9\" (UID: \"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59\") " pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:04.273453 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.272533 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb-run\") pod \"tuned-hzfnv\" (UID: \"2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-hzfnv" Sep 10 15:42:04.274039 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:04.272549 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbfbb1ad-2b61-4739-926f-01774a102c98-metrics-certs podName:bbfbb1ad-2b61-4739-926f-01774a102c98 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:04.772529416 +0000 UTC m=+2.126048574 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bbfbb1ad-2b61-4739-926f-01774a102c98-metrics-certs") pod "network-metrics-daemon-9d4jc" (UID: "bbfbb1ad-2b61-4739-926f-01774a102c98") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 15:42:04.274039 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.272577 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb-lib-modules\") pod \"tuned-hzfnv\" (UID: \"2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-hzfnv" Sep 10 15:42:04.274039 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.272582 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dd44752e-23f1-4ebe-a1ce-f7ad0561fc59-ovnkube-script-lib\") pod \"ovnkube-node-zc9b9\" (UID: \"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59\") " pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:04.274039 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.272591 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd44752e-23f1-4ebe-a1ce-f7ad0561fc59-var-lib-openvswitch\") pod \"ovnkube-node-zc9b9\" (UID: \"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59\") " pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:04.274039 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.272761 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb-sys\") pod \"tuned-hzfnv\" (UID: \"2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-hzfnv" Sep 10 15:42:04.274039 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.273055 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0fb1831f-d8df-47b6-aeb9-21565c4ffe9e-iptables-alerter-script\") pod \"iptables-alerter-xcbq4\" (UID: \"0fb1831f-d8df-47b6-aeb9-21565c4ffe9e\") " pod="openshift-network-operator/iptables-alerter-xcbq4" Sep 10 15:42:04.274039 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.273125 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd44752e-23f1-4ebe-a1ce-f7ad0561fc59-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zc9b9\" (UID: \"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59\") " pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:04.274039 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.273320 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb-etc-systemd\") pod \"tuned-hzfnv\" (UID: \"2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-hzfnv" Sep 10 15:42:04.274039 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.273361 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8nzf\" (UniqueName: \"kubernetes.io/projected/dd44752e-23f1-4ebe-a1ce-f7ad0561fc59-kube-api-access-j8nzf\") pod \"ovnkube-node-zc9b9\" (UID: \"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59\") " pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:04.274039 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.273392 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd44752e-23f1-4ebe-a1ce-f7ad0561fc59-host-kubelet\") pod \"ovnkube-node-zc9b9\" (UID: \"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59\") " pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:04.274039 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.273380 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dd44752e-23f1-4ebe-a1ce-f7ad0561fc59-ovnkube-script-lib\") pod \"ovnkube-node-zc9b9\" (UID: \"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59\") " pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:04.274039 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.273417 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb-etc-systemd\") pod \"tuned-hzfnv\" (UID: \"2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-hzfnv" Sep 10 15:42:04.274039 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.273465 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd44752e-23f1-4ebe-a1ce-f7ad0561fc59-host-kubelet\") pod \"ovnkube-node-zc9b9\" (UID: \"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59\") " pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:04.274677 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.274656 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb-tmp\") pod \"tuned-hzfnv\" (UID: \"2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-hzfnv" Sep 10 15:42:04.274831 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.274815 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dd44752e-23f1-4ebe-a1ce-f7ad0561fc59-ovn-node-metrics-cert\") pod \"ovnkube-node-zc9b9\" (UID: \"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59\") " pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:04.274916 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.274901 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb-etc-tuned\") pod \"tuned-hzfnv\" (UID: \"2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-hzfnv" Sep 10 15:42:04.281150 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.281094 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v8xd\" (UniqueName: \"kubernetes.io/projected/2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb-kube-api-access-2v8xd\") pod \"tuned-hzfnv\" (UID: \"2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb\") " pod="openshift-cluster-node-tuning-operator/tuned-hzfnv" Sep 10 15:42:04.281150 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.281114 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbvhl\" (UniqueName: \"kubernetes.io/projected/bbfbb1ad-2b61-4739-926f-01774a102c98-kube-api-access-kbvhl\") pod \"network-metrics-daemon-9d4jc\" (UID: \"bbfbb1ad-2b61-4739-926f-01774a102c98\") " pod="openshift-multus/network-metrics-daemon-9d4jc" Sep 10 15:42:04.281603 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:04.281581 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8043e4990f69187018df43e2af02f19.slice/crio-fdb1589fdaea90415dc9a20604f7af5f3206b93586051ed2c95e441a0f05e9ef WatchSource:0}: Error finding container fdb1589fdaea90415dc9a20604f7af5f3206b93586051ed2c95e441a0f05e9ef: Status 404 returned error can't find the container with id fdb1589fdaea90415dc9a20604f7af5f3206b93586051ed2c95e441a0f05e9ef Sep 10 15:42:04.282144 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.282115 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxq2x\" (UniqueName: \"kubernetes.io/projected/0fb1831f-d8df-47b6-aeb9-21565c4ffe9e-kube-api-access-wxq2x\") pod \"iptables-alerter-xcbq4\" (UID: \"0fb1831f-d8df-47b6-aeb9-21565c4ffe9e\") " pod="openshift-network-operator/iptables-alerter-xcbq4" Sep 10 15:42:04.284710 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.282583 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8nzf\" (UniqueName: \"kubernetes.io/projected/dd44752e-23f1-4ebe-a1ce-f7ad0561fc59-kube-api-access-j8nzf\") pod \"ovnkube-node-zc9b9\" (UID: \"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59\") " pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:04.286968 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.286954 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 15:42:04.381331 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.381236 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5sgqq" Sep 10 15:42:04.391444 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:04.391418 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2e3e295_41d8_4b2f_b0a4_26ec4ec7d11a.slice/crio-de66cdefc3764abaf1082ac8ffded9e7d0d0cfbbca12a8be55be841cb1eb6c35 WatchSource:0}: Error finding container de66cdefc3764abaf1082ac8ffded9e7d0d0cfbbca12a8be55be841cb1eb6c35: Status 404 returned error can't find the container with id de66cdefc3764abaf1082ac8ffded9e7d0d0cfbbca12a8be55be841cb1eb6c35 Sep 10 15:42:04.404921 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.404896 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d46d4" Sep 10 15:42:04.412840 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:04.412813 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50da34c6_e20d_41cf_8f08_95c7143e087c.slice/crio-1822b97617146f9e5301aa77164bf63d5aef5efa3fe66b3106317a9047d6a39f WatchSource:0}: Error finding container 1822b97617146f9e5301aa77164bf63d5aef5efa3fe66b3106317a9047d6a39f: Status 404 returned error can't find the container with id 1822b97617146f9e5301aa77164bf63d5aef5efa3fe66b3106317a9047d6a39f Sep 10 15:42:04.425101 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.425082 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5zp9c" Sep 10 15:42:04.427230 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.427209 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xwmfs" Sep 10 15:42:04.434622 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:04.434598 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b8c9295_101d_4700_a819_ae5172b0deee.slice/crio-cbd774aefb5effba4ac109ec2d204bcecf5043db1f9bac471e537d7bcc73999e WatchSource:0}: Error finding container cbd774aefb5effba4ac109ec2d204bcecf5043db1f9bac471e537d7bcc73999e: Status 404 returned error can't find the container with id cbd774aefb5effba4ac109ec2d204bcecf5043db1f9bac471e537d7bcc73999e Sep 10 15:42:04.436076 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:04.436009 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod322fb473_4c72_4194_9bba_9728743cdd74.slice/crio-80f3a62bbb5ca3278032633505ee4a909426af76d3ff911871972a92bc815ca3 WatchSource:0}: Error finding container 80f3a62bbb5ca3278032633505ee4a909426af76d3ff911871972a92bc815ca3: Status 404 returned error can't find the container with id 80f3a62bbb5ca3278032633505ee4a909426af76d3ff911871972a92bc815ca3 Sep 10 15:42:04.445603 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.445586 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5hkzp" Sep 10 15:42:04.454987 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:04.454958 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod849ac9ca_6d6b_4759_8871_2afe71e4afe3.slice/crio-34b431a77a93f48792336dffb11bc8f7cf6e0c1605df6cd4e1bc91b0b5c87404 WatchSource:0}: Error finding container 34b431a77a93f48792336dffb11bc8f7cf6e0c1605df6cd4e1bc91b0b5c87404: Status 404 returned error can't find the container with id 34b431a77a93f48792336dffb11bc8f7cf6e0c1605df6cd4e1bc91b0b5c87404 Sep 10 15:42:04.461669 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.461645 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-72k8k" Sep 10 15:42:04.469190 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:04.469173 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd17f9024_a71a_480f_acab_1f6e6279f055.slice/crio-7aa5070182c6d7888d23e1c710f8bb93da317edc1a2e5938571a90c347fb24ca WatchSource:0}: Error finding container 7aa5070182c6d7888d23e1c710f8bb93da317edc1a2e5938571a90c347fb24ca: Status 404 returned error can't find the container with id 7aa5070182c6d7888d23e1c710f8bb93da317edc1a2e5938571a90c347fb24ca Sep 10 15:42:04.489428 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.489406 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hzfnv" Sep 10 15:42:04.497957 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:04.497933 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e999b3a_b4d3_48c9_bbec_6ce8244e0dcb.slice/crio-2dca8b6dcbf9dafe0ddf0d44595af9af9174cc80398e3ade68e90dca766fa81e WatchSource:0}: Error finding container 2dca8b6dcbf9dafe0ddf0d44595af9af9174cc80398e3ade68e90dca766fa81e: Status 404 returned error can't find the container with id 2dca8b6dcbf9dafe0ddf0d44595af9af9174cc80398e3ade68e90dca766fa81e Sep 10 15:42:04.506282 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.506261 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xcbq4" Sep 10 15:42:04.511852 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.511838 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:04.514625 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:04.514560 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fb1831f_d8df_47b6_aeb9_21565c4ffe9e.slice/crio-ed0a76ea7712968d8aa3126bf46771e84579e3143a6fd12aaafaf1ee01b1266a WatchSource:0}: Error finding container ed0a76ea7712968d8aa3126bf46771e84579e3143a6fd12aaafaf1ee01b1266a: Status 404 returned error can't find the container with id ed0a76ea7712968d8aa3126bf46771e84579e3143a6fd12aaafaf1ee01b1266a Sep 10 15:42:04.519681 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:04.519654 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd44752e_23f1_4ebe_a1ce_f7ad0561fc59.slice/crio-71f06b62abca8db0c531d44feee62cad4ca4026dedade1a0f94ced097351bed0 WatchSource:0}: Error finding container 71f06b62abca8db0c531d44feee62cad4ca4026dedade1a0f94ced097351bed0: Status 404 returned error can't find the container with id 71f06b62abca8db0c531d44feee62cad4ca4026dedade1a0f94ced097351bed0 Sep 10 15:42:04.676147 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.676055 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1b1f777f-8ed4-408b-9721-80c9ae47af6e-original-pull-secret\") pod \"global-pull-secret-syncer-hzjqc\" (UID: \"1b1f777f-8ed4-408b-9721-80c9ae47af6e\") " pod="kube-system/global-pull-secret-syncer-hzjqc" Sep 10 15:42:04.676299 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:04.676185 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 15:42:04.676299 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:04.676243 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b1f777f-8ed4-408b-9721-80c9ae47af6e-original-pull-secret podName:1b1f777f-8ed4-408b-9721-80c9ae47af6e nodeName:}" failed. No retries permitted until 2026-09-10 15:42:05.676225311 +0000 UTC m=+3.029744453 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1b1f777f-8ed4-408b-9721-80c9ae47af6e-original-pull-secret") pod "global-pull-secret-syncer-hzjqc" (UID: "1b1f777f-8ed4-408b-9721-80c9ae47af6e") : object "kube-system"/"original-pull-secret" not registered Sep 10 15:42:04.776455 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.776423 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbfbb1ad-2b61-4739-926f-01774a102c98-metrics-certs\") pod \"network-metrics-daemon-9d4jc\" (UID: \"bbfbb1ad-2b61-4739-926f-01774a102c98\") " pod="openshift-multus/network-metrics-daemon-9d4jc" Sep 10 15:42:04.776650 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:04.776470 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppv8k\" (UniqueName: \"kubernetes.io/projected/c27ce7b1-1b71-4f92-8897-fd4748ebd262-kube-api-access-ppv8k\") pod \"network-check-target-ptjhb\" (UID: \"c27ce7b1-1b71-4f92-8897-fd4748ebd262\") " pod="openshift-network-diagnostics/network-check-target-ptjhb" Sep 10 15:42:04.776650 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:04.776610 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 15:42:04.776650 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:04.776607 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 15:42:04.776650 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:04.776628 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 15:42:04.776650 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:04.776644 2576 projected.go:196] Error preparing data for projected volume kube-api-access-ppv8k for pod openshift-network-diagnostics/network-check-target-ptjhb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 15:42:04.776981 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:04.776694 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbfbb1ad-2b61-4739-926f-01774a102c98-metrics-certs podName:bbfbb1ad-2b61-4739-926f-01774a102c98 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:05.776673305 +0000 UTC m=+3.130192461 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bbfbb1ad-2b61-4739-926f-01774a102c98-metrics-certs") pod "network-metrics-daemon-9d4jc" (UID: "bbfbb1ad-2b61-4739-926f-01774a102c98") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 15:42:04.776981 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:04.776715 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c27ce7b1-1b71-4f92-8897-fd4748ebd262-kube-api-access-ppv8k podName:c27ce7b1-1b71-4f92-8897-fd4748ebd262 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:05.776704846 +0000 UTC m=+3.130223996 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ppv8k" (UniqueName: "kubernetes.io/projected/c27ce7b1-1b71-4f92-8897-fd4748ebd262-kube-api-access-ppv8k") pod "network-check-target-ptjhb" (UID: "c27ce7b1-1b71-4f92-8897-fd4748ebd262") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 15:42:05.107767 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:05.107655 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 15:37:04 +0000 UTC" deadline="2028-05-30 14:44:43.424246792 +0000 UTC" Sep 10 15:42:05.107767 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:05.107720 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15071h2m38.316530976s" Sep 10 15:42:05.166660 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:05.166619 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 15:42:05.197628 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:05.197520 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-236.ec2.internal" event={"ID":"c089abe4aa32d6992a9466be84e92f1f","Type":"ContainerStarted","Data":"5f166719002e09cf42fa813e057884e8d3ca568ee4f3ec6c8830d7c3db3ca92d"} Sep 10 15:42:05.215863 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:05.215823 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-236.ec2.internal" event={"ID":"a8043e4990f69187018df43e2af02f19","Type":"ContainerStarted","Data":"fdb1589fdaea90415dc9a20604f7af5f3206b93586051ed2c95e441a0f05e9ef"} Sep 10 15:42:05.227555 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:05.224187 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xcbq4" event={"ID":"0fb1831f-d8df-47b6-aeb9-21565c4ffe9e","Type":"ContainerStarted","Data":"ed0a76ea7712968d8aa3126bf46771e84579e3143a6fd12aaafaf1ee01b1266a"} Sep 10 15:42:05.227555 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:05.226999 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hzfnv" event={"ID":"2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb","Type":"ContainerStarted","Data":"2dca8b6dcbf9dafe0ddf0d44595af9af9174cc80398e3ade68e90dca766fa81e"} Sep 10 15:42:05.234678 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:05.234528 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-72k8k" event={"ID":"d17f9024-a71a-480f-acab-1f6e6279f055","Type":"ContainerStarted","Data":"7aa5070182c6d7888d23e1c710f8bb93da317edc1a2e5938571a90c347fb24ca"} Sep 10 15:42:05.243847 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:05.243745 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5hkzp" event={"ID":"849ac9ca-6d6b-4759-8871-2afe71e4afe3","Type":"ContainerStarted","Data":"34b431a77a93f48792336dffb11bc8f7cf6e0c1605df6cd4e1bc91b0b5c87404"} Sep 10 15:42:05.249633 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:05.249518 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d46d4" event={"ID":"50da34c6-e20d-41cf-8f08-95c7143e087c","Type":"ContainerStarted","Data":"1822b97617146f9e5301aa77164bf63d5aef5efa3fe66b3106317a9047d6a39f"} Sep 10 15:42:05.264194 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:05.264159 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5sgqq" event={"ID":"a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a","Type":"ContainerStarted","Data":"de66cdefc3764abaf1082ac8ffded9e7d0d0cfbbca12a8be55be841cb1eb6c35"} Sep 10 15:42:05.273701 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:05.273670 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" event={"ID":"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59","Type":"ContainerStarted","Data":"71f06b62abca8db0c531d44feee62cad4ca4026dedade1a0f94ced097351bed0"} Sep 10 15:42:05.278663 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:05.278636 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xwmfs" event={"ID":"322fb473-4c72-4194-9bba-9728743cdd74","Type":"ContainerStarted","Data":"80f3a62bbb5ca3278032633505ee4a909426af76d3ff911871972a92bc815ca3"} Sep 10 15:42:05.300846 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:05.300810 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5zp9c" event={"ID":"3b8c9295-101d-4700-a819-ae5172b0deee","Type":"ContainerStarted","Data":"cbd774aefb5effba4ac109ec2d204bcecf5043db1f9bac471e537d7bcc73999e"} Sep 10 15:42:05.683933 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:05.683891 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1b1f777f-8ed4-408b-9721-80c9ae47af6e-original-pull-secret\") pod \"global-pull-secret-syncer-hzjqc\" (UID: \"1b1f777f-8ed4-408b-9721-80c9ae47af6e\") " pod="kube-system/global-pull-secret-syncer-hzjqc" Sep 10 15:42:05.684123 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:05.684039 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 15:42:05.684123 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:05.684102 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b1f777f-8ed4-408b-9721-80c9ae47af6e-original-pull-secret podName:1b1f777f-8ed4-408b-9721-80c9ae47af6e nodeName:}" failed. No retries permitted until 2026-09-10 15:42:07.684084919 +0000 UTC m=+5.037604076 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1b1f777f-8ed4-408b-9721-80c9ae47af6e-original-pull-secret") pod "global-pull-secret-syncer-hzjqc" (UID: "1b1f777f-8ed4-408b-9721-80c9ae47af6e") : object "kube-system"/"original-pull-secret" not registered Sep 10 15:42:05.784712 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:05.784671 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbfbb1ad-2b61-4739-926f-01774a102c98-metrics-certs\") pod \"network-metrics-daemon-9d4jc\" (UID: \"bbfbb1ad-2b61-4739-926f-01774a102c98\") " pod="openshift-multus/network-metrics-daemon-9d4jc" Sep 10 15:42:05.784940 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:05.784732 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppv8k\" (UniqueName: \"kubernetes.io/projected/c27ce7b1-1b71-4f92-8897-fd4748ebd262-kube-api-access-ppv8k\") pod \"network-check-target-ptjhb\" (UID: \"c27ce7b1-1b71-4f92-8897-fd4748ebd262\") " pod="openshift-network-diagnostics/network-check-target-ptjhb" Sep 10 15:42:05.785011 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:05.784946 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 15:42:05.785011 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:05.784969 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 15:42:05.785011 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:05.784983 2576 projected.go:196] Error preparing data for projected volume kube-api-access-ppv8k for pod openshift-network-diagnostics/network-check-target-ptjhb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 15:42:05.785170 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:05.785044 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c27ce7b1-1b71-4f92-8897-fd4748ebd262-kube-api-access-ppv8k podName:c27ce7b1-1b71-4f92-8897-fd4748ebd262 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:07.785024852 +0000 UTC m=+5.138544002 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ppv8k" (UniqueName: "kubernetes.io/projected/c27ce7b1-1b71-4f92-8897-fd4748ebd262-kube-api-access-ppv8k") pod "network-check-target-ptjhb" (UID: "c27ce7b1-1b71-4f92-8897-fd4748ebd262") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 15:42:05.785474 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:05.785452 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 15:42:05.785555 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:05.785510 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbfbb1ad-2b61-4739-926f-01774a102c98-metrics-certs podName:bbfbb1ad-2b61-4739-926f-01774a102c98 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:07.785494409 +0000 UTC m=+5.139013558 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bbfbb1ad-2b61-4739-926f-01774a102c98-metrics-certs") pod "network-metrics-daemon-9d4jc" (UID: "bbfbb1ad-2b61-4739-926f-01774a102c98") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 15:42:06.108255 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:06.108164 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 15:37:04 +0000 UTC" deadline="2028-02-21 08:57:06.854071721 +0000 UTC" Sep 10 15:42:06.108255 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:06.108207 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12689h15m0.745870111s" Sep 10 15:42:06.163150 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:06.163107 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9d4jc" Sep 10 15:42:06.163339 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:06.163279 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9d4jc" podUID="bbfbb1ad-2b61-4739-926f-01774a102c98" Sep 10 15:42:06.163771 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:06.163721 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ptjhb" Sep 10 15:42:06.163900 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:06.163863 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ptjhb" podUID="c27ce7b1-1b71-4f92-8897-fd4748ebd262" Sep 10 15:42:06.163996 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:06.163947 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hzjqc" Sep 10 15:42:06.164085 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:06.164038 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hzjqc" podUID="1b1f777f-8ed4-408b-9721-80c9ae47af6e" Sep 10 15:42:06.884104 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:06.884014 2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 15:42:07.703328 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:07.702672 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1b1f777f-8ed4-408b-9721-80c9ae47af6e-original-pull-secret\") pod \"global-pull-secret-syncer-hzjqc\" (UID: \"1b1f777f-8ed4-408b-9721-80c9ae47af6e\") " pod="kube-system/global-pull-secret-syncer-hzjqc" Sep 10 15:42:07.703328 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:07.702845 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 15:42:07.703328 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:07.702919 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b1f777f-8ed4-408b-9721-80c9ae47af6e-original-pull-secret podName:1b1f777f-8ed4-408b-9721-80c9ae47af6e nodeName:}" failed. No retries permitted until 2026-09-10 15:42:11.702900949 +0000 UTC m=+9.056420094 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1b1f777f-8ed4-408b-9721-80c9ae47af6e-original-pull-secret") pod "global-pull-secret-syncer-hzjqc" (UID: "1b1f777f-8ed4-408b-9721-80c9ae47af6e") : object "kube-system"/"original-pull-secret" not registered Sep 10 15:42:07.803340 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:07.803302 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbfbb1ad-2b61-4739-926f-01774a102c98-metrics-certs\") pod \"network-metrics-daemon-9d4jc\" (UID: \"bbfbb1ad-2b61-4739-926f-01774a102c98\") " pod="openshift-multus/network-metrics-daemon-9d4jc" Sep 10 15:42:07.803529 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:07.803355 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppv8k\" (UniqueName: \"kubernetes.io/projected/c27ce7b1-1b71-4f92-8897-fd4748ebd262-kube-api-access-ppv8k\") pod \"network-check-target-ptjhb\" (UID: \"c27ce7b1-1b71-4f92-8897-fd4748ebd262\") " pod="openshift-network-diagnostics/network-check-target-ptjhb" Sep 10 15:42:07.803529 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:07.803524 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 15:42:07.803658 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:07.803553 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 15:42:07.803658 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:07.803572 2576 projected.go:196] Error preparing data for projected volume kube-api-access-ppv8k for pod openshift-network-diagnostics/network-check-target-ptjhb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 15:42:07.803658 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:07.803623 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 15:42:07.803847 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:07.803636 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c27ce7b1-1b71-4f92-8897-fd4748ebd262-kube-api-access-ppv8k podName:c27ce7b1-1b71-4f92-8897-fd4748ebd262 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:11.803616402 +0000 UTC m=+9.157135570 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ppv8k" (UniqueName: "kubernetes.io/projected/c27ce7b1-1b71-4f92-8897-fd4748ebd262-kube-api-access-ppv8k") pod "network-check-target-ptjhb" (UID: "c27ce7b1-1b71-4f92-8897-fd4748ebd262") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 15:42:07.803847 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:07.803707 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbfbb1ad-2b61-4739-926f-01774a102c98-metrics-certs podName:bbfbb1ad-2b61-4739-926f-01774a102c98 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:11.803687805 +0000 UTC m=+9.157206953 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bbfbb1ad-2b61-4739-926f-01774a102c98-metrics-certs") pod "network-metrics-daemon-9d4jc" (UID: "bbfbb1ad-2b61-4739-926f-01774a102c98") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 15:42:08.162926 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:08.162833 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9d4jc" Sep 10 15:42:08.163107 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:08.162988 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9d4jc" podUID="bbfbb1ad-2b61-4739-926f-01774a102c98" Sep 10 15:42:08.163693 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:08.163392 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ptjhb" Sep 10 15:42:08.163693 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:08.163497 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ptjhb" podUID="c27ce7b1-1b71-4f92-8897-fd4748ebd262" Sep 10 15:42:08.163693 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:08.163572 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hzjqc" Sep 10 15:42:08.163693 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:08.163647 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hzjqc" podUID="1b1f777f-8ed4-408b-9721-80c9ae47af6e" Sep 10 15:42:10.163183 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:10.163140 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hzjqc" Sep 10 15:42:10.163685 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:10.163275 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hzjqc" podUID="1b1f777f-8ed4-408b-9721-80c9ae47af6e" Sep 10 15:42:10.163685 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:10.163679 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9d4jc" Sep 10 15:42:10.163855 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:10.163819 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9d4jc" podUID="bbfbb1ad-2b61-4739-926f-01774a102c98" Sep 10 15:42:10.163931 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:10.163874 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ptjhb" Sep 10 15:42:10.163984 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:10.163953 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ptjhb" podUID="c27ce7b1-1b71-4f92-8897-fd4748ebd262" Sep 10 15:42:11.739520 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:11.739466 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1b1f777f-8ed4-408b-9721-80c9ae47af6e-original-pull-secret\") pod \"global-pull-secret-syncer-hzjqc\" (UID: \"1b1f777f-8ed4-408b-9721-80c9ae47af6e\") " pod="kube-system/global-pull-secret-syncer-hzjqc" Sep 10 15:42:11.740015 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:11.739649 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 15:42:11.740015 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:11.739738 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b1f777f-8ed4-408b-9721-80c9ae47af6e-original-pull-secret podName:1b1f777f-8ed4-408b-9721-80c9ae47af6e nodeName:}" failed. No retries permitted until 2026-09-10 15:42:19.739713933 +0000 UTC m=+17.093233077 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1b1f777f-8ed4-408b-9721-80c9ae47af6e-original-pull-secret") pod "global-pull-secret-syncer-hzjqc" (UID: "1b1f777f-8ed4-408b-9721-80c9ae47af6e") : object "kube-system"/"original-pull-secret" not registered Sep 10 15:42:11.840751 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:11.840705 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbfbb1ad-2b61-4739-926f-01774a102c98-metrics-certs\") pod \"network-metrics-daemon-9d4jc\" (UID: \"bbfbb1ad-2b61-4739-926f-01774a102c98\") " pod="openshift-multus/network-metrics-daemon-9d4jc" Sep 10 15:42:11.840967 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:11.840779 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppv8k\" (UniqueName: \"kubernetes.io/projected/c27ce7b1-1b71-4f92-8897-fd4748ebd262-kube-api-access-ppv8k\") pod \"network-check-target-ptjhb\" (UID: \"c27ce7b1-1b71-4f92-8897-fd4748ebd262\") " pod="openshift-network-diagnostics/network-check-target-ptjhb" Sep 10 15:42:11.840967 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:11.840868 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 15:42:11.840967 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:11.840940 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbfbb1ad-2b61-4739-926f-01774a102c98-metrics-certs podName:bbfbb1ad-2b61-4739-926f-01774a102c98 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:19.840924278 +0000 UTC m=+17.194443420 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bbfbb1ad-2b61-4739-926f-01774a102c98-metrics-certs") pod "network-metrics-daemon-9d4jc" (UID: "bbfbb1ad-2b61-4739-926f-01774a102c98") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 15:42:11.840967 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:11.840959 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 15:42:11.841216 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:11.840987 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 15:42:11.841216 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:11.841001 2576 projected.go:196] Error preparing data for projected volume kube-api-access-ppv8k for pod openshift-network-diagnostics/network-check-target-ptjhb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 15:42:11.841216 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:11.841073 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c27ce7b1-1b71-4f92-8897-fd4748ebd262-kube-api-access-ppv8k podName:c27ce7b1-1b71-4f92-8897-fd4748ebd262 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:19.841054652 +0000 UTC m=+17.194573812 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ppv8k" (UniqueName: "kubernetes.io/projected/c27ce7b1-1b71-4f92-8897-fd4748ebd262-kube-api-access-ppv8k") pod "network-check-target-ptjhb" (UID: "c27ce7b1-1b71-4f92-8897-fd4748ebd262") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 15:42:12.163817 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:12.163225 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9d4jc" Sep 10 15:42:12.163817 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:12.163225 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ptjhb" Sep 10 15:42:12.163817 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:12.163356 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9d4jc" podUID="bbfbb1ad-2b61-4739-926f-01774a102c98" Sep 10 15:42:12.163817 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:12.163236 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hzjqc" Sep 10 15:42:12.163817 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:12.163428 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ptjhb" podUID="c27ce7b1-1b71-4f92-8897-fd4748ebd262" Sep 10 15:42:12.163817 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:12.163516 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hzjqc" podUID="1b1f777f-8ed4-408b-9721-80c9ae47af6e" Sep 10 15:42:14.163556 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:14.163461 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hzjqc" Sep 10 15:42:14.164085 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:14.163597 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hzjqc" podUID="1b1f777f-8ed4-408b-9721-80c9ae47af6e" Sep 10 15:42:14.164085 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:14.164067 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9d4jc" Sep 10 15:42:14.164208 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:14.164184 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9d4jc" podUID="bbfbb1ad-2b61-4739-926f-01774a102c98" Sep 10 15:42:14.164269 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:14.164246 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ptjhb" Sep 10 15:42:14.164386 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:14.164322 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ptjhb" podUID="c27ce7b1-1b71-4f92-8897-fd4748ebd262" Sep 10 15:42:16.162812 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:16.162775 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hzjqc" Sep 10 15:42:16.163420 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:16.162822 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9d4jc" Sep 10 15:42:16.163420 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:16.162906 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hzjqc" podUID="1b1f777f-8ed4-408b-9721-80c9ae47af6e" Sep 10 15:42:16.163420 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:16.162976 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9d4jc" podUID="bbfbb1ad-2b61-4739-926f-01774a102c98" Sep 10 15:42:16.163420 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:16.162985 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ptjhb" Sep 10 15:42:16.163420 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:16.163073 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ptjhb" podUID="c27ce7b1-1b71-4f92-8897-fd4748ebd262" Sep 10 15:42:18.162684 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:18.162653 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hzjqc" Sep 10 15:42:18.163149 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:18.162662 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ptjhb" Sep 10 15:42:18.163149 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:18.162786 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hzjqc" podUID="1b1f777f-8ed4-408b-9721-80c9ae47af6e" Sep 10 15:42:18.163149 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:18.162664 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9d4jc" Sep 10 15:42:18.163149 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:18.162847 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ptjhb" podUID="c27ce7b1-1b71-4f92-8897-fd4748ebd262" Sep 10 15:42:18.163149 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:18.162958 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9d4jc" podUID="bbfbb1ad-2b61-4739-926f-01774a102c98" Sep 10 15:42:19.803168 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:19.803137 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1b1f777f-8ed4-408b-9721-80c9ae47af6e-original-pull-secret\") pod \"global-pull-secret-syncer-hzjqc\" (UID: \"1b1f777f-8ed4-408b-9721-80c9ae47af6e\") " pod="kube-system/global-pull-secret-syncer-hzjqc" Sep 10 15:42:19.803628 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:19.803283 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 15:42:19.803628 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:19.803354 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b1f777f-8ed4-408b-9721-80c9ae47af6e-original-pull-secret podName:1b1f777f-8ed4-408b-9721-80c9ae47af6e nodeName:}" failed. No retries permitted until 2026-09-10 15:42:35.803333007 +0000 UTC m=+33.156852170 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1b1f777f-8ed4-408b-9721-80c9ae47af6e-original-pull-secret") pod "global-pull-secret-syncer-hzjqc" (UID: "1b1f777f-8ed4-408b-9721-80c9ae47af6e") : object "kube-system"/"original-pull-secret" not registered Sep 10 15:42:19.903488 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:19.903451 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbfbb1ad-2b61-4739-926f-01774a102c98-metrics-certs\") pod \"network-metrics-daemon-9d4jc\" (UID: \"bbfbb1ad-2b61-4739-926f-01774a102c98\") " pod="openshift-multus/network-metrics-daemon-9d4jc" Sep 10 15:42:19.903488 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:19.903496 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppv8k\" (UniqueName: \"kubernetes.io/projected/c27ce7b1-1b71-4f92-8897-fd4748ebd262-kube-api-access-ppv8k\") pod \"network-check-target-ptjhb\" (UID: \"c27ce7b1-1b71-4f92-8897-fd4748ebd262\") " pod="openshift-network-diagnostics/network-check-target-ptjhb" Sep 10 15:42:19.903761 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:19.903613 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 15:42:19.903761 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:19.903642 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 15:42:19.903761 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:19.903655 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 15:42:19.903761 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:19.903663 2576 projected.go:196] Error preparing data for projected volume kube-api-access-ppv8k for pod openshift-network-diagnostics/network-check-target-ptjhb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 15:42:19.903761 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:19.903679 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbfbb1ad-2b61-4739-926f-01774a102c98-metrics-certs podName:bbfbb1ad-2b61-4739-926f-01774a102c98 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:35.903661785 +0000 UTC m=+33.257180934 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bbfbb1ad-2b61-4739-926f-01774a102c98-metrics-certs") pod "network-metrics-daemon-9d4jc" (UID: "bbfbb1ad-2b61-4739-926f-01774a102c98") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 15:42:19.903761 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:19.903705 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c27ce7b1-1b71-4f92-8897-fd4748ebd262-kube-api-access-ppv8k podName:c27ce7b1-1b71-4f92-8897-fd4748ebd262 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:35.903695122 +0000 UTC m=+33.257214262 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ppv8k" (UniqueName: "kubernetes.io/projected/c27ce7b1-1b71-4f92-8897-fd4748ebd262-kube-api-access-ppv8k") pod "network-check-target-ptjhb" (UID: "c27ce7b1-1b71-4f92-8897-fd4748ebd262") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 15:42:20.162857 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:20.162780 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ptjhb" Sep 10 15:42:20.162857 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:20.162811 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hzjqc" Sep 10 15:42:20.162857 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:20.162826 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9d4jc" Sep 10 15:42:20.163153 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:20.162909 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ptjhb" podUID="c27ce7b1-1b71-4f92-8897-fd4748ebd262" Sep 10 15:42:20.163153 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:20.163052 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hzjqc" podUID="1b1f777f-8ed4-408b-9721-80c9ae47af6e" Sep 10 15:42:20.163232 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:20.163185 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9d4jc" podUID="bbfbb1ad-2b61-4739-926f-01774a102c98" Sep 10 15:42:22.163028 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:22.162999 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ptjhb" Sep 10 15:42:22.163704 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:22.163100 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hzjqc" Sep 10 15:42:22.163704 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:22.163111 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ptjhb" podUID="c27ce7b1-1b71-4f92-8897-fd4748ebd262" Sep 10 15:42:22.163704 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:22.163132 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9d4jc" Sep 10 15:42:22.163704 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:22.163213 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hzjqc" podUID="1b1f777f-8ed4-408b-9721-80c9ae47af6e" Sep 10 15:42:22.163704 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:22.163267 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9d4jc" podUID="bbfbb1ad-2b61-4739-926f-01774a102c98" Sep 10 15:42:24.163479 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:24.163264 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ptjhb" Sep 10 15:42:24.163479 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:24.163368 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hzjqc" Sep 10 15:42:24.163479 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:24.163384 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ptjhb" podUID="c27ce7b1-1b71-4f92-8897-fd4748ebd262" Sep 10 15:42:24.163479 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:24.163404 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9d4jc" Sep 10 15:42:24.163981 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:24.163498 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hzjqc" podUID="1b1f777f-8ed4-408b-9721-80c9ae47af6e" Sep 10 15:42:24.163981 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:24.163579 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9d4jc" podUID="bbfbb1ad-2b61-4739-926f-01774a102c98" Sep 10 15:42:25.349131 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:25.348809 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-236.ec2.internal" event={"ID":"c089abe4aa32d6992a9466be84e92f1f","Type":"ContainerStarted","Data":"a9e70dd17a7520d9bbc750256f70a51287d28223c449cb75b2faad5c58e43667"} Sep 10 15:42:25.350593 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:25.350562 2576 generic.go:356] "Generic (PLEG): container finished" podID="a8043e4990f69187018df43e2af02f19" containerID="5020c0578f2328d7c7f7933b0738c6514a068c9d027fb084bec6ab66ce870515" exitCode=0 Sep 10 15:42:25.350708 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:25.350645 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-236.ec2.internal" event={"ID":"a8043e4990f69187018df43e2af02f19","Type":"ContainerDied","Data":"5020c0578f2328d7c7f7933b0738c6514a068c9d027fb084bec6ab66ce870515"} Sep 10 15:42:25.352545 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:25.352516 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hzfnv" event={"ID":"2e999b3a-b4d3-48c9-bbec-6ce8244e0dcb","Type":"ContainerStarted","Data":"8ac89099f79a60ea691718e3dffcee3baa505f6547025b64e95513b60036dd2f"} Sep 10 15:42:25.354314 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:25.354295 2576 generic.go:356] "Generic (PLEG): container finished" podID="d17f9024-a71a-480f-acab-1f6e6279f055" containerID="3351fd888cc20b8ebc6835929e50be35590a419d2c64207f159a21d0a1b52116" exitCode=0 Sep 10 15:42:25.354408 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:25.354325 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-72k8k" event={"ID":"d17f9024-a71a-480f-acab-1f6e6279f055","Type":"ContainerDied","Data":"3351fd888cc20b8ebc6835929e50be35590a419d2c64207f159a21d0a1b52116"} Sep 10 15:42:25.356169 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:25.356148 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5hkzp" event={"ID":"849ac9ca-6d6b-4759-8871-2afe71e4afe3","Type":"ContainerStarted","Data":"fbd6f16cb79324e4ee4362a021ad6228563a920eea9e08fc5bdb9815de68319a"} Sep 10 15:42:25.357782 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:25.357762 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d46d4" event={"ID":"50da34c6-e20d-41cf-8f08-95c7143e087c","Type":"ContainerStarted","Data":"e151cd249c15a4553740f00fa868e43934fd01ccaf13e93593e494767d3132ec"} Sep 10 15:42:25.359304 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:25.359283 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5sgqq" event={"ID":"a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a","Type":"ContainerStarted","Data":"9e99512c2e26fb5fd23ee40e18bf56ff6549ce24f29d7f49ca3aa853e99e3fd7"} Sep 10 15:42:25.362629 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:25.362581 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-236.ec2.internal" podStartSLOduration=21.362566052 podStartE2EDuration="21.362566052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:42:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:25.362245108 +0000 UTC m=+22.715764483" watchObservedRunningTime="2026-09-10 15:42:25.362566052 +0000 UTC m=+22.716085215" Sep 10 15:42:25.362743 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:25.362652 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" event={"ID":"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59","Type":"ContainerStarted","Data":"7122dcae349f8d19a051457f5f34ff818b13c366e4d1a40ba69633493248dd5f"} Sep 10 15:42:25.362799 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:25.362725 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" event={"ID":"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59","Type":"ContainerStarted","Data":"327bd8ec232626c2dfb3ea9d94598e2b60282171532ac77468caef5e67bf5887"} Sep 10 15:42:25.362844 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:25.362813 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" event={"ID":"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59","Type":"ContainerStarted","Data":"c48c76e1737ec250115fe79a7c1d86a23de81f768dcb444fe1a251c3cef209c3"} Sep 10 15:42:25.362883 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:25.362864 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" event={"ID":"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59","Type":"ContainerStarted","Data":"9f47a51e09dc6f67388bafae92ac86b2d197229f6805a3e9584b1a81cf92e31f"} Sep 10 15:42:25.362932 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:25.362901 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" event={"ID":"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59","Type":"ContainerStarted","Data":"91a70eae8d8dc09edabcc17f0fa8a3edf8fea9ec787a5c438bca137b66067ca9"} Sep 10 15:42:25.362980 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:25.362941 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" event={"ID":"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59","Type":"ContainerStarted","Data":"d9c8d8e04fc26857ada3770d141c20d5cfcf9134a6873ba808c9ab2f91ccba2e"} Sep 10 15:42:25.364194 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:25.364162 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xwmfs" event={"ID":"322fb473-4c72-4194-9bba-9728743cdd74","Type":"ContainerStarted","Data":"5564e6479fa4b3e656e2503f3079e19b2edf04ef52bd5873d0b7ee4aed91705d"} Sep 10 15:42:25.365553 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:25.365534 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5zp9c" event={"ID":"3b8c9295-101d-4700-a819-ae5172b0deee","Type":"ContainerStarted","Data":"eb92e7613863252e2c6ac021ea379deb819add6886c8970432ff2f5f384797c3"} Sep 10 15:42:25.395704 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:25.394378 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-5hkzp" podStartSLOduration=1.238549206 podStartE2EDuration="21.394361741s" totalImagesPullingTime="20.155812535s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:42:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:25.394090678 +0000 UTC m=+22.747609839" watchObservedRunningTime="2026-09-10 15:42:25.394361741 +0000 UTC m=+22.747880908" Sep 10 15:42:25.407727 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:25.407694 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-5sgqq" podStartSLOduration=1.4354547420000001 podStartE2EDuration="21.407682151s" totalImagesPullingTime="19.972227409s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:42:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:25.40762331 +0000 UTC m=+22.761142489" watchObservedRunningTime="2026-09-10 15:42:25.407682151 +0000 UTC m=+22.761201310" Sep 10 15:42:25.444896 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:25.444861 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-hzfnv" podStartSLOduration=1.383589031 podStartE2EDuration="21.444850424s" totalImagesPullingTime="20.061261393s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:42:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:25.444782094 +0000 UTC m=+22.798301271" watchObservedRunningTime="2026-09-10 15:42:25.444850424 +0000 UTC m=+22.798369586" Sep 10 15:42:25.463178 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:25.463144 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-xwmfs" podStartSLOduration=1.352170338 podStartE2EDuration="21.463132862s" totalImagesPullingTime="20.110962524s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:42:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:25.463013799 +0000 UTC m=+22.816532961" watchObservedRunningTime="2026-09-10 15:42:25.463132862 +0000 UTC m=+22.816652024" Sep 10 15:42:25.479041 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:25.478999 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-5zp9c" podStartSLOduration=1.366415125 podStartE2EDuration="21.478982792s" totalImagesPullingTime="20.112567667s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:42:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:25.478304121 +0000 UTC m=+22.831823282" watchObservedRunningTime="2026-09-10 15:42:25.478982792 +0000 UTC m=+22.832501955" Sep 10 15:42:25.520686 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:25.520661 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:25.575547 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:25.575519 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:26.162515 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:26.162489 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9d4jc" Sep 10 15:42:26.162627 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:26.162502 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ptjhb" Sep 10 15:42:26.162627 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:26.162594 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hzjqc" Sep 10 15:42:26.162627 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:26.162591 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9d4jc" podUID="bbfbb1ad-2b61-4739-926f-01774a102c98" Sep 10 15:42:26.162726 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:26.162656 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ptjhb" podUID="c27ce7b1-1b71-4f92-8897-fd4748ebd262" Sep 10 15:42:26.162779 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:26.162750 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hzjqc" podUID="1b1f777f-8ed4-408b-9721-80c9ae47af6e" Sep 10 15:42:26.371199 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:26.371169 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-236.ec2.internal" event={"ID":"a8043e4990f69187018df43e2af02f19","Type":"ContainerStarted","Data":"14255bbde51bfbd7a6c50c53024c0d0943d80bf491e6b8128803ec43ebc9f799"} Sep 10 15:42:26.372665 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:26.372641 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xcbq4" event={"ID":"0fb1831f-d8df-47b6-aeb9-21565c4ffe9e","Type":"ContainerStarted","Data":"61d51cbefc5aabaccd4d202a51fff00c9b66efe9b9220434132fe0ec3b4c5e90"} Sep 10 15:42:26.400779 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:26.400725 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-236.ec2.internal" podStartSLOduration=22.400712877 podStartE2EDuration="22.400712877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:42:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:26.385865168 +0000 UTC m=+23.739384331" watchObservedRunningTime="2026-09-10 15:42:26.400712877 +0000 UTC m=+23.754232039" Sep 10 15:42:26.552483 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:26.552457 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:26.614286 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:26.614261 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:27.035987 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:27.035819 2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 10 15:42:27.129876 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:27.129780 2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T15:42:27.035997527Z","UUID":"1622ad09-7059-423a-9078-fd00a0d98e35","Handler":null,"Name":"","Endpoint":""} Sep 10 15:42:27.131557 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:27.131536 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 10 15:42:27.131661 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:27.131598 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 10 15:42:27.376922 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:27.376880 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d46d4" event={"ID":"50da34c6-e20d-41cf-8f08-95c7143e087c","Type":"ContainerStarted","Data":"ea4e43e46d0f1485bf42a5f26ff442398aa051bb80423be47eeec8a20634631a"} Sep 10 15:42:27.847458 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:27.847428 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-5sgqq" Sep 10 15:42:27.848235 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:27.848213 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-5sgqq" Sep 10 15:42:27.863974 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:27.863930 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xcbq4" podStartSLOduration=3.826926684 podStartE2EDuration="23.863917071s" totalImagesPullingTime="20.036990387s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:42:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:26.400847948 +0000 UTC m=+23.754367088" watchObservedRunningTime="2026-09-10 15:42:27.863917071 +0000 UTC m=+25.217436235" Sep 10 15:42:28.163431 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:28.163354 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ptjhb" Sep 10 15:42:28.163431 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:28.163359 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9d4jc" Sep 10 15:42:28.163631 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:28.163464 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hzjqc" Sep 10 15:42:28.163631 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:28.163462 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ptjhb" podUID="c27ce7b1-1b71-4f92-8897-fd4748ebd262" Sep 10 15:42:28.163631 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:28.163559 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9d4jc" podUID="bbfbb1ad-2b61-4739-926f-01774a102c98" Sep 10 15:42:28.163791 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:28.163632 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hzjqc" podUID="1b1f777f-8ed4-408b-9721-80c9ae47af6e" Sep 10 15:42:28.383492 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:28.383453 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" event={"ID":"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59","Type":"ContainerStarted","Data":"945e203ed5e58669159f619dc28fb8f5dae348a10146cfcc5971354e36a4212c"} Sep 10 15:42:29.225109 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:29.225061 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:29.322991 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:29.322958 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-5sgqq" Sep 10 15:42:29.323669 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:29.323644 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5sgqq" Sep 10 15:42:29.336086 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:29.336059 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:29.434195 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:29.434167 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:30.163414 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:30.163375 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hzjqc" Sep 10 15:42:30.163593 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:30.163510 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ptjhb" Sep 10 15:42:30.163593 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:30.163521 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9d4jc" Sep 10 15:42:30.163593 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:30.163512 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hzjqc" podUID="1b1f777f-8ed4-408b-9721-80c9ae47af6e" Sep 10 15:42:30.163787 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:30.163595 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ptjhb" podUID="c27ce7b1-1b71-4f92-8897-fd4748ebd262" Sep 10 15:42:30.163787 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:30.163679 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9d4jc" podUID="bbfbb1ad-2b61-4739-926f-01774a102c98" Sep 10 15:42:30.285210 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:30.285160 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:30.389343 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:30.389309 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:31.392431 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:31.392203 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d46d4" event={"ID":"50da34c6-e20d-41cf-8f08-95c7143e087c","Type":"ContainerStarted","Data":"aa6ae8f66bdc8e7ffcc889a7218799c9c1f9a406b00d805e678fd66bbfab2ede"} Sep 10 15:42:31.396398 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:31.396368 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" event={"ID":"dd44752e-23f1-4ebe-a1ce-f7ad0561fc59","Type":"ContainerStarted","Data":"bfbdf82e37df5bf3e65961cca6296b8ee11b85fd0771d6f535c469e3182b0a1b"} Sep 10 15:42:31.396683 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:31.396659 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:31.396809 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:31.396745 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:31.398516 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:31.398492 2576 generic.go:356] "Generic (PLEG): container finished" podID="d17f9024-a71a-480f-acab-1f6e6279f055" containerID="9d0e62415f8d3d1d194998ae159f89bbf440c574d5faf3a119ae712d9a4288ab" exitCode=0 Sep 10 15:42:31.398697 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:31.398527 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-72k8k" event={"ID":"d17f9024-a71a-480f-acab-1f6e6279f055","Type":"ContainerDied","Data":"9d0e62415f8d3d1d194998ae159f89bbf440c574d5faf3a119ae712d9a4288ab"} Sep 10 15:42:31.409426 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:31.409386 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d46d4" podStartSLOduration=1.455298486 podStartE2EDuration="27.409373058s" totalImagesPullingTime="25.954074572s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:42:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:31.409176078 +0000 UTC m=+28.762695344" watchObservedRunningTime="2026-09-10 15:42:31.409373058 +0000 UTC m=+28.762892263" Sep 10 15:42:31.413185 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:31.413163 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:31.447719 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:31.447654 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" podStartSLOduration=7.35536011 podStartE2EDuration="27.447642795s" totalImagesPullingTime="20.092282685s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:42:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:31.446969116 +0000 UTC m=+28.800488268" watchObservedRunningTime="2026-09-10 15:42:31.447642795 +0000 UTC m=+28.801161957" Sep 10 15:42:31.943615 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:31.943585 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:32.012120 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:32.012084 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:32.162786 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:32.162681 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9d4jc" Sep 10 15:42:32.162786 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:32.162702 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hzjqc" Sep 10 15:42:32.162786 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:32.162687 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ptjhb" Sep 10 15:42:32.163044 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:32.162850 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9d4jc" podUID="bbfbb1ad-2b61-4739-926f-01774a102c98" Sep 10 15:42:32.163044 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:32.162960 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hzjqc" podUID="1b1f777f-8ed4-408b-9721-80c9ae47af6e" Sep 10 15:42:32.163044 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:32.163035 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ptjhb" podUID="c27ce7b1-1b71-4f92-8897-fd4748ebd262" Sep 10 15:42:32.177602 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:32.177567 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:32.287337 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:32.287302 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:32.397463 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:32.397428 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:32.401638 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:32.401612 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:32.420115 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:32.420048 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:42:32.496257 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:32.496227 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:32.602499 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:32.602470 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:32.882042 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:32.882003 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:32.944218 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:32.944194 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:32.975940 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:32.975916 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hzjqc"] Sep 10 15:42:32.976027 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:32.976001 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hzjqc" Sep 10 15:42:32.976102 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:32.976082 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hzjqc" podUID="1b1f777f-8ed4-408b-9721-80c9ae47af6e" Sep 10 15:42:32.980439 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:32.979804 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ptjhb"] Sep 10 15:42:32.980439 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:32.979877 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ptjhb" Sep 10 15:42:32.980439 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:32.979966 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ptjhb" podUID="c27ce7b1-1b71-4f92-8897-fd4748ebd262" Sep 10 15:42:32.980439 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:32.980342 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9d4jc"] Sep 10 15:42:32.980439 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:32.980409 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9d4jc" Sep 10 15:42:32.980676 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:32.980509 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9d4jc" podUID="bbfbb1ad-2b61-4739-926f-01774a102c98" Sep 10 15:42:33.304548 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:33.304520 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5zp9c_3b8c9295-101d-4700-a819-ae5172b0deee/dns-node-resolver/0.log" Sep 10 15:42:33.405724 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:33.405693 2576 generic.go:356] "Generic (PLEG): container finished" podID="d17f9024-a71a-480f-acab-1f6e6279f055" containerID="4435f5fc15e773f26c12cbc01c869bc2e76c0ed3366af6f4c8b43f8d9a366d93" exitCode=0 Sep 10 15:42:33.405724 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:33.405764 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-72k8k" event={"ID":"d17f9024-a71a-480f-acab-1f6e6279f055","Type":"ContainerDied","Data":"4435f5fc15e773f26c12cbc01c869bc2e76c0ed3366af6f4c8b43f8d9a366d93"} Sep 10 15:42:34.162778 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:34.162753 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ptjhb" Sep 10 15:42:34.162928 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:34.162851 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ptjhb" podUID="c27ce7b1-1b71-4f92-8897-fd4748ebd262" Sep 10 15:42:34.274764 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:34.274703 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:34.301563 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:34.301539 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xwmfs_322fb473-4c72-4194-9bba-9728743cdd74/node-ca/0.log" Sep 10 15:42:34.376317 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:34.376284 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:34.409999 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:34.409971 2576 generic.go:356] "Generic (PLEG): container finished" podID="d17f9024-a71a-480f-acab-1f6e6279f055" containerID="55f783a7090d91af7436ab56c08e2e97e01a1a56bc45bbebdec4c428c605030c" exitCode=0 Sep 10 15:42:34.410320 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:34.410047 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-72k8k" event={"ID":"d17f9024-a71a-480f-acab-1f6e6279f055","Type":"ContainerDied","Data":"55f783a7090d91af7436ab56c08e2e97e01a1a56bc45bbebdec4c428c605030c"} Sep 10 15:42:34.508277 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:34.508251 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:34.612036 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:34.612015 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:35.141284 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:35.141246 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:35.163146 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:35.163120 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9d4jc" Sep 10 15:42:35.163278 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:35.163240 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9d4jc" podUID="bbfbb1ad-2b61-4739-926f-01774a102c98" Sep 10 15:42:35.163355 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:35.163289 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hzjqc" Sep 10 15:42:35.163410 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:35.163378 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hzjqc" podUID="1b1f777f-8ed4-408b-9721-80c9ae47af6e" Sep 10 15:42:35.818977 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:35.818936 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1b1f777f-8ed4-408b-9721-80c9ae47af6e-original-pull-secret\") pod \"global-pull-secret-syncer-hzjqc\" (UID: \"1b1f777f-8ed4-408b-9721-80c9ae47af6e\") " pod="kube-system/global-pull-secret-syncer-hzjqc" Sep 10 15:42:35.819425 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:35.819106 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 15:42:35.819425 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:35.819185 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b1f777f-8ed4-408b-9721-80c9ae47af6e-original-pull-secret podName:1b1f777f-8ed4-408b-9721-80c9ae47af6e nodeName:}" failed. No retries permitted until 2026-09-10 15:43:07.819165346 +0000 UTC m=+65.172684507 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1b1f777f-8ed4-408b-9721-80c9ae47af6e-original-pull-secret") pod "global-pull-secret-syncer-hzjqc" (UID: "1b1f777f-8ed4-408b-9721-80c9ae47af6e") : object "kube-system"/"original-pull-secret" not registered Sep 10 15:42:35.919538 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:35.919505 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbfbb1ad-2b61-4739-926f-01774a102c98-metrics-certs\") pod \"network-metrics-daemon-9d4jc\" (UID: \"bbfbb1ad-2b61-4739-926f-01774a102c98\") " pod="openshift-multus/network-metrics-daemon-9d4jc" Sep 10 15:42:35.919677 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:35.919550 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppv8k\" (UniqueName: \"kubernetes.io/projected/c27ce7b1-1b71-4f92-8897-fd4748ebd262-kube-api-access-ppv8k\") pod \"network-check-target-ptjhb\" (UID: \"c27ce7b1-1b71-4f92-8897-fd4748ebd262\") " pod="openshift-network-diagnostics/network-check-target-ptjhb" Sep 10 15:42:35.919677 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:35.919647 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 15:42:35.919677 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:35.919672 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 15:42:35.919830 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:35.919692 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 15:42:35.919830 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:35.919705 2576 projected.go:196] Error preparing data for projected volume kube-api-access-ppv8k for pod openshift-network-diagnostics/network-check-target-ptjhb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 15:42:35.919830 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:35.919712 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbfbb1ad-2b61-4739-926f-01774a102c98-metrics-certs podName:bbfbb1ad-2b61-4739-926f-01774a102c98 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:07.919692913 +0000 UTC m=+65.273212070 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bbfbb1ad-2b61-4739-926f-01774a102c98-metrics-certs") pod "network-metrics-daemon-9d4jc" (UID: "bbfbb1ad-2b61-4739-926f-01774a102c98") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 15:42:35.919830 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:35.919771 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c27ce7b1-1b71-4f92-8897-fd4748ebd262-kube-api-access-ppv8k podName:c27ce7b1-1b71-4f92-8897-fd4748ebd262 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:07.919755746 +0000 UTC m=+65.273274894 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-ppv8k" (UniqueName: "kubernetes.io/projected/c27ce7b1-1b71-4f92-8897-fd4748ebd262-kube-api-access-ppv8k") pod "network-check-target-ptjhb" (UID: "c27ce7b1-1b71-4f92-8897-fd4748ebd262") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 15:42:36.018362 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:36.018304 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:36.097553 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:36.097482 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:36.162931 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:36.162906 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ptjhb" Sep 10 15:42:36.163064 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:36.163007 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ptjhb" podUID="c27ce7b1-1b71-4f92-8897-fd4748ebd262" Sep 10 15:42:36.320640 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:36.320553 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:36.740097 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:36.740014 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:36.812377 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:36.812349 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:36.998889 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:36.998822 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:37.094463 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:37.094434 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:37.162858 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:37.162828 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9d4jc" Sep 10 15:42:37.163019 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:37.162829 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hzjqc" Sep 10 15:42:37.163019 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:37.162959 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9d4jc" podUID="bbfbb1ad-2b61-4739-926f-01774a102c98" Sep 10 15:42:37.163121 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:37.163048 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hzjqc" podUID="1b1f777f-8ed4-408b-9721-80c9ae47af6e" Sep 10 15:42:37.197688 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:37.197659 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:37.257749 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:37.257661 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:37.429743 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:37.429690 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:38.163097 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:38.163064 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ptjhb" Sep 10 15:42:38.163547 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:38.163183 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ptjhb" podUID="c27ce7b1-1b71-4f92-8897-fd4748ebd262" Sep 10 15:42:38.227142 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:38.227108 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:38.626377 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:38.626350 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:38.728273 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:38.728242 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:39.163353 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:39.163304 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9d4jc" Sep 10 15:42:39.163353 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:39.163344 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hzjqc" Sep 10 15:42:39.163747 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:39.163465 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9d4jc" podUID="bbfbb1ad-2b61-4739-926f-01774a102c98" Sep 10 15:42:39.163747 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:39.163605 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hzjqc" podUID="1b1f777f-8ed4-408b-9721-80c9ae47af6e" Sep 10 15:42:39.365943 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:39.365910 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:39.427349 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:39.427287 2576 generic.go:356] "Generic (PLEG): container finished" podID="d17f9024-a71a-480f-acab-1f6e6279f055" containerID="38c71ac3983c82efa2d728dfe62ada52929d83a959689ac05164cbc1230ef602" exitCode=0 Sep 10 15:42:39.427349 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:39.427327 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-72k8k" event={"ID":"d17f9024-a71a-480f-acab-1f6e6279f055","Type":"ContainerDied","Data":"38c71ac3983c82efa2d728dfe62ada52929d83a959689ac05164cbc1230ef602"} Sep 10 15:42:39.451781 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:39.451601 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:39.831264 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:39.831231 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:40.035093 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:40.035058 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:40.162874 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:40.162814 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ptjhb" Sep 10 15:42:40.162985 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:40.162897 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ptjhb" podUID="c27ce7b1-1b71-4f92-8897-fd4748ebd262" Sep 10 15:42:40.203157 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:40.203095 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:40.342709 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:40.342686 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:40.433821 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:40.433763 2576 generic.go:356] "Generic (PLEG): container finished" podID="d17f9024-a71a-480f-acab-1f6e6279f055" containerID="b3a54b3ff16cdd3bd0f31358b951bb232a4f34d4e2ce4b57021bfe83ed45519b" exitCode=0 Sep 10 15:42:40.433945 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:40.433823 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-72k8k" event={"ID":"d17f9024-a71a-480f-acab-1f6e6279f055","Type":"ContainerDied","Data":"b3a54b3ff16cdd3bd0f31358b951bb232a4f34d4e2ce4b57021bfe83ed45519b"} Sep 10 15:42:40.899505 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:40.899467 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:41.163098 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:41.163042 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hzjqc" Sep 10 15:42:41.163211 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:41.163130 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hzjqc" podUID="1b1f777f-8ed4-408b-9721-80c9ae47af6e" Sep 10 15:42:41.163257 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:41.163230 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9d4jc" Sep 10 15:42:41.163362 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:41.163339 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9d4jc" podUID="bbfbb1ad-2b61-4739-926f-01774a102c98" Sep 10 15:42:41.230165 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:41.230142 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:41.292401 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:41.292370 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:41.439300 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:41.439244 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-72k8k" event={"ID":"d17f9024-a71a-480f-acab-1f6e6279f055","Type":"ContainerStarted","Data":"8df3fccc7cefdc3573326751f4c237c553d8d376c98d4cc413c2bb0dc1338a46"} Sep 10 15:42:41.461082 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:41.461047 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-72k8k" podStartSLOduration=5.718829605 podStartE2EDuration="37.461034655s" totalImagesPullingTime="31.74220505s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:42:04 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:41.460764165 +0000 UTC m=+38.814283348" watchObservedRunningTime="2026-09-10 15:42:41.461034655 +0000 UTC m=+38.814553816" Sep 10 15:42:41.628794 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:41.628763 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:41.825969 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:41.825941 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:41.925988 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:41.925965 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:42.026346 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:42.026308 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:42.080856 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:42.080797 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:42.163038 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:42.163015 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ptjhb" Sep 10 15:42:42.163128 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:42.163092 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ptjhb" podUID="c27ce7b1-1b71-4f92-8897-fd4748ebd262" Sep 10 15:42:42.432175 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:42.432104 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:42.788259 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:42.788190 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:42.886657 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:42.886627 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:43.163326 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:43.163302 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9d4jc" Sep 10 15:42:43.163475 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:43.163413 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9d4jc" podUID="bbfbb1ad-2b61-4739-926f-01774a102c98" Sep 10 15:42:43.163547 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:43.163507 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hzjqc" Sep 10 15:42:43.163626 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:43.163605 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hzjqc" podUID="1b1f777f-8ed4-408b-9721-80c9ae47af6e" Sep 10 15:42:43.228202 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:43.228173 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:43.621489 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:43.621458 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:43.825066 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:43.825039 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:44.026600 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:44.026533 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:44.162560 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:44.162537 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ptjhb" Sep 10 15:42:44.162661 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:44.162622 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ptjhb" podUID="c27ce7b1-1b71-4f92-8897-fd4748ebd262" Sep 10 15:42:44.226168 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:44.226138 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:44.442664 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:44.442624 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:45.105491 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:45.105453 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:45.162969 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:45.162943 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hzjqc" Sep 10 15:42:45.163110 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:45.163030 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hzjqc" podUID="1b1f777f-8ed4-408b-9721-80c9ae47af6e" Sep 10 15:42:45.163110 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:45.163057 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9d4jc" Sep 10 15:42:45.163227 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:45.163113 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9d4jc" podUID="bbfbb1ad-2b61-4739-926f-01774a102c98" Sep 10 15:42:45.225569 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:45.225540 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:45.626358 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:45.626329 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:45.822181 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:45.822147 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:46.020189 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:46.020123 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:46.163124 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:46.163100 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ptjhb" Sep 10 15:42:46.163448 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:46.163192 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ptjhb" podUID="c27ce7b1-1b71-4f92-8897-fd4748ebd262" Sep 10 15:42:46.384972 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:46.384946 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:46.456318 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:46.456285 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:46.828660 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:46.828623 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:47.031234 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:47.031199 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:47.138281 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:47.138213 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:47.163375 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:47.163349 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9d4jc" Sep 10 15:42:47.163375 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:47.163369 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hzjqc" Sep 10 15:42:47.163707 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:47.163443 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9d4jc" podUID="bbfbb1ad-2b61-4739-926f-01774a102c98" Sep 10 15:42:47.163707 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:47.163561 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hzjqc" podUID="1b1f777f-8ed4-408b-9721-80c9ae47af6e" Sep 10 15:42:47.193228 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:47.193208 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:48.061525 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:48.061492 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:48.163011 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:48.162986 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ptjhb" Sep 10 15:42:48.163129 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:48.163070 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ptjhb" podUID="c27ce7b1-1b71-4f92-8897-fd4748ebd262" Sep 10 15:42:48.884722 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:48.884689 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:48.989835 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:48.989798 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:49.088909 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:49.088877 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:49.163119 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:49.163052 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hzjqc" Sep 10 15:42:49.163227 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:49.163146 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hzjqc" podUID="1b1f777f-8ed4-408b-9721-80c9ae47af6e" Sep 10 15:42:49.163285 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:49.163225 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9d4jc" Sep 10 15:42:49.163345 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:49.163328 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9d4jc" podUID="bbfbb1ad-2b61-4739-926f-01774a102c98" Sep 10 15:42:49.206859 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:49.206835 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:49.573639 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:49.573609 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:49.635792 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:49.635766 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:49.775924 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:49.775893 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:49.837943 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:49.837890 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:49.973397 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:49.973363 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:50.028945 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:50.028922 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:50.163205 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:50.163156 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ptjhb" Sep 10 15:42:50.163281 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:50.163238 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ptjhb" podUID="c27ce7b1-1b71-4f92-8897-fd4748ebd262" Sep 10 15:42:50.379572 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:50.379548 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:50.577706 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:50.577686 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:50.698088 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:50.698063 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:50.784882 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:50.784855 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:50.993940 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:50.993874 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:51.097768 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:51.097727 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:51.162048 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:51.162025 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:51.162960 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:51.162944 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9d4jc" Sep 10 15:42:51.163045 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:51.162955 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hzjqc" Sep 10 15:42:51.163108 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:51.163090 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9d4jc" podUID="bbfbb1ad-2b61-4739-926f-01774a102c98" Sep 10 15:42:51.163206 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:51.163186 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hzjqc" podUID="1b1f777f-8ed4-408b-9721-80c9ae47af6e" Sep 10 15:42:51.603134 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:51.603098 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:51.790891 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:51.790864 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:52.162759 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:52.162722 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ptjhb" Sep 10 15:42:52.163128 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:52.162846 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ptjhb" podUID="c27ce7b1-1b71-4f92-8897-fd4748ebd262" Sep 10 15:42:53.163312 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:53.163276 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hzjqc" Sep 10 15:42:53.163683 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:53.163366 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hzjqc" podUID="1b1f777f-8ed4-408b-9721-80c9ae47af6e" Sep 10 15:42:53.163683 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:53.163443 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9d4jc" Sep 10 15:42:53.163683 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:53.163539 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9d4jc" podUID="bbfbb1ad-2b61-4739-926f-01774a102c98" Sep 10 15:42:54.163218 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:54.163182 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ptjhb" Sep 10 15:42:54.163409 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:54.163283 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ptjhb" podUID="c27ce7b1-1b71-4f92-8897-fd4748ebd262" Sep 10 15:42:55.162810 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:55.162778 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9d4jc" Sep 10 15:42:55.162998 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:55.162784 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hzjqc" Sep 10 15:42:55.162998 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:55.162892 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9d4jc" podUID="bbfbb1ad-2b61-4739-926f-01774a102c98" Sep 10 15:42:55.162998 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:55.162945 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hzjqc" podUID="1b1f777f-8ed4-408b-9721-80c9ae47af6e" Sep 10 15:42:55.183975 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:55.183938 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:55.301620 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:55.301590 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:55.401118 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:55.401095 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:55.502661 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:55.502589 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:55.613623 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:55.613601 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:55.737768 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:55.737727 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:56.162952 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:56.162928 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ptjhb" Sep 10 15:42:56.163051 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:42:56.163019 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ptjhb" podUID="c27ce7b1-1b71-4f92-8897-fd4748ebd262" Sep 10 15:42:56.286543 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:56.286517 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:56.390343 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:56.390292 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:56.429605 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:56.429587 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-236.ec2.internal" event="NodeReady" Sep 10 15:42:56.429676 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:56.429670 2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 10 15:42:56.501005 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:56.500982 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-nt5ps"] Sep 10 15:42:56.503759 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:56.503721 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nt5ps" Sep 10 15:42:56.504690 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:56.504672 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-wqh2d"] Sep 10 15:42:56.505967 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:56.505951 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-brxch\"" Sep 10 15:42:56.506186 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:56.506173 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 10 15:42:56.509399 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:56.509381 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wqh2d" Sep 10 15:42:56.509492 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:56.509444 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 10 15:42:56.511888 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:56.511869 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 10 15:42:56.512335 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:56.512316 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 10 15:42:56.512335 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:56.512325 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 10 15:42:56.512574 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:56.512557 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5dvnv\"" Sep 10 15:42:56.512662 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:56.512648 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 10 15:42:56.516514 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:56.516494 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nt5ps"] Sep 10 15:42:56.527646 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:56.527624 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8rhvj"] Sep 10 15:42:56.530296 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:56.530280 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8rhvj" Sep 10 15:42:56.530749 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:56.530710 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wqh2d"] Sep 10 15:42:56.533558 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:56.533543 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 10 15:42:56.533873 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:56.533855 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 10 15:42:56.533984 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:56.533953 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-pdj5g\"" Sep 10 15:42:56.534046 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:56.533986 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 10 15:42:56.540904 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:56.540887 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8rhvj"] Sep 10 15:42:56.649586 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:56.649518 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:56.669380 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:56.669362 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0d22a485-8c50-428d-8da7-8ca4e831c2c8-data-volume\") pod \"insights-runtime-extractor-wqh2d\" (UID: \"0d22a485-8c50-428d-8da7-8ca4e831c2c8\") " pod="openshift-insights/insights-runtime-extractor-wqh2d" Sep 10 15:42:56.669440 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:56.669390 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qbzt\" (UniqueName: \"kubernetes.io/projected/93cdfcd5-7e56-4acf-ac1b-eb9896ba6581-kube-api-access-8qbzt\") pod \"dns-default-nt5ps\" (UID: \"93cdfcd5-7e56-4acf-ac1b-eb9896ba6581\") " pod="openshift-dns/dns-default-nt5ps" Sep 10 15:42:56.669440 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:56.669412 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpwx6\" (UniqueName: \"kubernetes.io/projected/9191d5ea-e8a5-4674-86cd-05d2c6949141-kube-api-access-mpwx6\") pod \"ingress-canary-8rhvj\" (UID: \"9191d5ea-e8a5-4674-86cd-05d2c6949141\") " pod="openshift-ingress-canary/ingress-canary-8rhvj" Sep 10 15:42:56.669505 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:56.669437 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/93cdfcd5-7e56-4acf-ac1b-eb9896ba6581-config-volume\") pod \"dns-default-nt5ps\" (UID: \"93cdfcd5-7e56-4acf-ac1b-eb9896ba6581\") " pod="openshift-dns/dns-default-nt5ps" Sep 10 15:42:56.669505 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:56.669485 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h44d4\" (UniqueName: \"kubernetes.io/projected/0d22a485-8c50-428d-8da7-8ca4e831c2c8-kube-api-access-h44d4\") pod \"insights-runtime-extractor-wqh2d\" (UID: \"0d22a485-8c50-428d-8da7-8ca4e831c2c8\") " pod="openshift-insights/insights-runtime-extractor-wqh2d" Sep 10 15:42:56.669505 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:56.669504 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9191d5ea-e8a5-4674-86cd-05d2c6949141-cert\") pod \"ingress-canary-8rhvj\" (UID: \"9191d5ea-e8a5-4674-86cd-05d2c6949141\") " pod="openshift-ingress-canary/ingress-canary-8rhvj" Sep 10 15:42:56.669594 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:56.669563 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0d22a485-8c50-428d-8da7-8ca4e831c2c8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wqh2d\" (UID: \"0d22a485-8c50-428d-8da7-8ca4e831c2c8\") " pod="openshift-insights/insights-runtime-extractor-wqh2d" Sep 10 15:42:56.669594 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:56.669588 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93cdfcd5-7e56-4acf-ac1b-eb9896ba6581-tmp-dir\") pod \"dns-default-nt5ps\" (UID: \"93cdfcd5-7e56-4acf-ac1b-eb9896ba6581\") " pod="openshift-dns/dns-default-nt5ps" Sep 10 15:42:56.669652 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:56.669613 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/93cdfcd5-7e56-4acf-ac1b-eb9896ba6581-metrics-tls\") pod \"dns-default-nt5ps\" (UID: \"93cdfcd5-7e56-4acf-ac1b-eb9896ba6581\") " pod="openshift-dns/dns-default-nt5ps" Sep 10 15:42:56.669652 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:56.669627 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0d22a485-8c50-428d-8da7-8ca4e831c2c8-crio-socket\") pod \"insights-runtime-extractor-wqh2d\" (UID: \"0d22a485-8c50-428d-8da7-8ca4e831c2c8\") " pod="openshift-insights/insights-runtime-extractor-wqh2d" Sep 10 15:42:56.669711 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:56.669658 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0d22a485-8c50-428d-8da7-8ca4e831c2c8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wqh2d\" (UID: \"0d22a485-8c50-428d-8da7-8ca4e831c2c8\") " pod="openshift-insights/insights-runtime-extractor-wqh2d" Sep 10 15:42:56.711710 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:56.711673 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:56.770632 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:56.770616 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/93cdfcd5-7e56-4acf-ac1b-eb9896ba6581-config-volume\") pod \"dns-default-nt5ps\" (UID: \"93cdfcd5-7e56-4acf-ac1b-eb9896ba6581\") " pod="openshift-dns/dns-default-nt5ps" Sep 10 15:42:56.770709 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:56.770648 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h44d4\" (UniqueName: \"kubernetes.io/projected/0d22a485-8c50-428d-8da7-8ca4e831c2c8-kube-api-access-h44d4\") pod \"insights-runtime-extractor-wqh2d\" (UID: \"0d22a485-8c50-428d-8da7-8ca4e831c2c8\") " pod="openshift-insights/insights-runtime-extractor-wqh2d" Sep 10 15:42:56.770709 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:56.770672 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9191d5ea-e8a5-4674-86cd-05d2c6949141-cert\") pod \"ingress-canary-8rhvj\" (UID: \"9191d5ea-e8a5-4674-86cd-05d2c6949141\") " pod="openshift-ingress-canary/ingress-canary-8rhvj" Sep 10 15:42:56.770804 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:56.770719 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0d22a485-8c50-428d-8da7-8ca4e831c2c8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wqh2d\" (UID: \"0d22a485-8c50-428d-8da7-8ca4e831c2c8\") " pod="openshift-insights/insights-runtime-extractor-wqh2d" Sep 10 15:42:56.770804 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:56.770764 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93cdfcd5-7e56-4acf-ac1b-eb9896ba6581-tmp-dir\") pod \"dns-default-nt5ps\" (UID: \"93cdfcd5-7e56-4acf-ac1b-eb9896ba6581\") " pod="openshift-dns/dns-default-nt5ps" Sep 10 15:42:56.770804 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:56.770789 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/93cdfcd5-7e56-4acf-ac1b-eb9896ba6581-metrics-tls\") pod \"dns-default-nt5ps\" (UID: \"93cdfcd5-7e56-4acf-ac1b-eb9896ba6581\") " pod="openshift-dns/dns-default-nt5ps" Sep 10 15:42:56.770946 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:56.770809 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0d22a485-8c50-428d-8da7-8ca4e831c2c8-crio-socket\") pod \"insights-runtime-extractor-wqh2d\" (UID: \"0d22a485-8c50-428d-8da7-8ca4e831c2c8\") " pod="openshift-insights/insights-runtime-extractor-wqh2d" Sep 10 15:42:56.770946 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:56.770844 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0d22a485-8c50-428d-8da7-8ca4e831c2c8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wqh2d\" (UID: \"0d22a485-8c50-428d-8da7-8ca4e831c2c8\") " pod="openshift-insights/insights-runtime-extractor-wqh2d" Sep 10 15:42:56.770946 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:56.770870 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0d22a485-8c50-428d-8da7-8ca4e831c2c8-data-volume\") pod \"insights-runtime-extractor-wqh2d\" (UID: \"0d22a485-8c50-428d-8da7-8ca4e831c2c8\") " pod="openshift-insights/insights-runtime-extractor-wqh2d" Sep 10 15:42:56.770946 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:56.770898 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qbzt\" (UniqueName: \"kubernetes.io/projected/93cdfcd5-7e56-4acf-ac1b-eb9896ba6581-kube-api-access-8qbzt\") pod \"dns-default-nt5ps\" (UID: \"93cdfcd5-7e56-4acf-ac1b-eb9896ba6581\") " pod="openshift-dns/dns-default-nt5ps" Sep 10 15:42:56.770946 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:56.770925 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpwx6\" (UniqueName: \"kubernetes.io/projected/9191d5ea-e8a5-4674-86cd-05d2c6949141-kube-api-access-mpwx6\") pod \"ingress-canary-8rhvj\" (UID: \"9191d5ea-e8a5-4674-86cd-05d2c6949141\") " pod="openshift-ingress-canary/ingress-canary-8rhvj" Sep 10 15:42:56.771182 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:56.771054 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0d22a485-8c50-428d-8da7-8ca4e831c2c8-crio-socket\") pod \"insights-runtime-extractor-wqh2d\" (UID: \"0d22a485-8c50-428d-8da7-8ca4e831c2c8\") " pod="openshift-insights/insights-runtime-extractor-wqh2d" Sep 10 15:42:56.771182 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:56.771075 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93cdfcd5-7e56-4acf-ac1b-eb9896ba6581-tmp-dir\") pod \"dns-default-nt5ps\" (UID: \"93cdfcd5-7e56-4acf-ac1b-eb9896ba6581\") " pod="openshift-dns/dns-default-nt5ps" Sep 10 15:42:56.771301 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:56.771181 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0d22a485-8c50-428d-8da7-8ca4e831c2c8-data-volume\") pod \"insights-runtime-extractor-wqh2d\" (UID: \"0d22a485-8c50-428d-8da7-8ca4e831c2c8\") " pod="openshift-insights/insights-runtime-extractor-wqh2d" Sep 10 15:42:56.771301 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:56.771272 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/93cdfcd5-7e56-4acf-ac1b-eb9896ba6581-config-volume\") pod \"dns-default-nt5ps\" (UID: \"93cdfcd5-7e56-4acf-ac1b-eb9896ba6581\") " pod="openshift-dns/dns-default-nt5ps" Sep 10 15:42:56.771598 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:56.771564 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0d22a485-8c50-428d-8da7-8ca4e831c2c8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wqh2d\" (UID: \"0d22a485-8c50-428d-8da7-8ca4e831c2c8\") " pod="openshift-insights/insights-runtime-extractor-wqh2d" Sep 10 15:42:56.774782 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:56.774757 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/93cdfcd5-7e56-4acf-ac1b-eb9896ba6581-metrics-tls\") pod \"dns-default-nt5ps\" (UID: \"93cdfcd5-7e56-4acf-ac1b-eb9896ba6581\") " pod="openshift-dns/dns-default-nt5ps" Sep 10 15:42:56.774782 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:56.774767 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0d22a485-8c50-428d-8da7-8ca4e831c2c8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wqh2d\" (UID: \"0d22a485-8c50-428d-8da7-8ca4e831c2c8\") " pod="openshift-insights/insights-runtime-extractor-wqh2d" Sep 10 15:42:56.774890 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:56.774807 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9191d5ea-e8a5-4674-86cd-05d2c6949141-cert\") pod \"ingress-canary-8rhvj\" (UID: \"9191d5ea-e8a5-4674-86cd-05d2c6949141\") " pod="openshift-ingress-canary/ingress-canary-8rhvj" Sep 10 15:42:56.777960 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:56.777935 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h44d4\" (UniqueName: \"kubernetes.io/projected/0d22a485-8c50-428d-8da7-8ca4e831c2c8-kube-api-access-h44d4\") pod \"insights-runtime-extractor-wqh2d\" (UID: \"0d22a485-8c50-428d-8da7-8ca4e831c2c8\") " pod="openshift-insights/insights-runtime-extractor-wqh2d" Sep 10 15:42:56.778122 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:56.778104 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpwx6\" (UniqueName: \"kubernetes.io/projected/9191d5ea-e8a5-4674-86cd-05d2c6949141-kube-api-access-mpwx6\") pod \"ingress-canary-8rhvj\" (UID: \"9191d5ea-e8a5-4674-86cd-05d2c6949141\") " pod="openshift-ingress-canary/ingress-canary-8rhvj" Sep 10 15:42:56.778764 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:56.778721 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qbzt\" (UniqueName: \"kubernetes.io/projected/93cdfcd5-7e56-4acf-ac1b-eb9896ba6581-kube-api-access-8qbzt\") pod \"dns-default-nt5ps\" (UID: \"93cdfcd5-7e56-4acf-ac1b-eb9896ba6581\") " pod="openshift-dns/dns-default-nt5ps" Sep 10 15:42:56.817633 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:56.817613 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nt5ps" Sep 10 15:42:56.821353 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:56.821332 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wqh2d" Sep 10 15:42:56.838971 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:56.838947 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8rhvj" Sep 10 15:42:57.103888 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:57.103856 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nt5ps"] Sep 10 15:42:57.107250 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:57.107219 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wqh2d"] Sep 10 15:42:57.107449 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:57.107427 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93cdfcd5_7e56_4acf_ac1b_eb9896ba6581.slice/crio-a79b1ac289b6fa4a80fc23ed732e325847e150ce2ebca35c54acd25ca6b33b18 WatchSource:0}: Error finding container a79b1ac289b6fa4a80fc23ed732e325847e150ce2ebca35c54acd25ca6b33b18: Status 404 returned error can't find the container with id a79b1ac289b6fa4a80fc23ed732e325847e150ce2ebca35c54acd25ca6b33b18 Sep 10 15:42:57.109818 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:57.109792 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d22a485_8c50_428d_8da7_8ca4e831c2c8.slice/crio-314aa2f518ce37833f414c43226841b2e3bf956d3c35e84d0745bad607b461a8 WatchSource:0}: Error finding container 314aa2f518ce37833f414c43226841b2e3bf956d3c35e84d0745bad607b461a8: Status 404 returned error can't find the container with id 314aa2f518ce37833f414c43226841b2e3bf956d3c35e84d0745bad607b461a8 Sep 10 15:42:57.130408 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:57.130382 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8rhvj"] Sep 10 15:42:57.133586 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:42:57.133561 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9191d5ea_e8a5_4674_86cd_05d2c6949141.slice/crio-4032bee580849661b0eea004ae55e5a3b27a7362e9ee4383b11a591030aecec0 WatchSource:0}: Error finding container 4032bee580849661b0eea004ae55e5a3b27a7362e9ee4383b11a591030aecec0: Status 404 returned error can't find the container with id 4032bee580849661b0eea004ae55e5a3b27a7362e9ee4383b11a591030aecec0 Sep 10 15:42:57.163482 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:57.163440 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9d4jc" Sep 10 15:42:57.163567 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:57.163441 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hzjqc" Sep 10 15:42:57.165688 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:57.165669 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2gdj6\"" Sep 10 15:42:57.165688 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:57.165677 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 10 15:42:57.165947 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:57.165934 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 10 15:42:57.472564 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:57.472492 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8rhvj" event={"ID":"9191d5ea-e8a5-4674-86cd-05d2c6949141","Type":"ContainerStarted","Data":"4032bee580849661b0eea004ae55e5a3b27a7362e9ee4383b11a591030aecec0"} Sep 10 15:42:57.474149 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:57.474113 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wqh2d" event={"ID":"0d22a485-8c50-428d-8da7-8ca4e831c2c8","Type":"ContainerStarted","Data":"cbac5b7db164219da4f43ac3b50f01bc6073927d72f7af2bc86344c13612850e"} Sep 10 15:42:57.474292 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:57.474169 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wqh2d" event={"ID":"0d22a485-8c50-428d-8da7-8ca4e831c2c8","Type":"ContainerStarted","Data":"314aa2f518ce37833f414c43226841b2e3bf956d3c35e84d0745bad607b461a8"} Sep 10 15:42:57.475444 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:57.475420 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nt5ps" event={"ID":"93cdfcd5-7e56-4acf-ac1b-eb9896ba6581","Type":"ContainerStarted","Data":"a79b1ac289b6fa4a80fc23ed732e325847e150ce2ebca35c54acd25ca6b33b18"} Sep 10 15:42:57.999980 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:57.999952 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d67c6549f-t9kz4"] Sep 10 15:42:58.007249 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:58.006499 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d67c6549f-t9kz4" Sep 10 15:42:58.007710 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:58.007680 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:58.008946 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:58.008921 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 10 15:42:58.009048 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:58.008931 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 10 15:42:58.009048 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:58.008950 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 10 15:42:58.010139 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:58.009602 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 10 15:42:58.010139 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:58.009665 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 10 15:42:58.010139 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:58.009669 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 10 15:42:58.010139 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:58.009770 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 10 15:42:58.010139 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:58.010015 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-mkfbn\"" Sep 10 15:42:58.012850 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:58.012830 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d67c6549f-t9kz4"] Sep 10 15:42:58.014704 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:58.014555 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 10 15:42:58.086629 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:58.086548 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:58.163267 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:58.163242 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ptjhb" Sep 10 15:42:58.165986 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:58.165960 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sjjtv\"" Sep 10 15:42:58.166089 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:58.166000 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 10 15:42:58.166089 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:58.165960 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 10 15:42:58.184074 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:58.184046 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af48efd8-d70a-4bc3-9f4f-250ce098c8e6-trusted-ca-bundle\") pod \"console-5d67c6549f-t9kz4\" (UID: \"af48efd8-d70a-4bc3-9f4f-250ce098c8e6\") " pod="openshift-console/console-5d67c6549f-t9kz4" Sep 10 15:42:58.184170 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:58.184100 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmt4v\" (UniqueName: \"kubernetes.io/projected/af48efd8-d70a-4bc3-9f4f-250ce098c8e6-kube-api-access-tmt4v\") pod \"console-5d67c6549f-t9kz4\" (UID: \"af48efd8-d70a-4bc3-9f4f-250ce098c8e6\") " pod="openshift-console/console-5d67c6549f-t9kz4" Sep 10 15:42:58.184170 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:58.184163 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/af48efd8-d70a-4bc3-9f4f-250ce098c8e6-console-serving-cert\") pod \"console-5d67c6549f-t9kz4\" (UID: \"af48efd8-d70a-4bc3-9f4f-250ce098c8e6\") " pod="openshift-console/console-5d67c6549f-t9kz4" Sep 10 15:42:58.184275 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:58.184201 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/af48efd8-d70a-4bc3-9f4f-250ce098c8e6-console-config\") pod \"console-5d67c6549f-t9kz4\" (UID: \"af48efd8-d70a-4bc3-9f4f-250ce098c8e6\") " pod="openshift-console/console-5d67c6549f-t9kz4" Sep 10 15:42:58.184275 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:58.184239 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/af48efd8-d70a-4bc3-9f4f-250ce098c8e6-service-ca\") pod \"console-5d67c6549f-t9kz4\" (UID: \"af48efd8-d70a-4bc3-9f4f-250ce098c8e6\") " pod="openshift-console/console-5d67c6549f-t9kz4" Sep 10 15:42:58.184363 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:58.184288 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/af48efd8-d70a-4bc3-9f4f-250ce098c8e6-console-oauth-config\") pod \"console-5d67c6549f-t9kz4\" (UID: \"af48efd8-d70a-4bc3-9f4f-250ce098c8e6\") " pod="openshift-console/console-5d67c6549f-t9kz4" Sep 10 15:42:58.184409 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:58.184383 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/af48efd8-d70a-4bc3-9f4f-250ce098c8e6-oauth-serving-cert\") pod \"console-5d67c6549f-t9kz4\" (UID: \"af48efd8-d70a-4bc3-9f4f-250ce098c8e6\") " pod="openshift-console/console-5d67c6549f-t9kz4" Sep 10 15:42:58.285481 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:58.285447 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmt4v\" (UniqueName: \"kubernetes.io/projected/af48efd8-d70a-4bc3-9f4f-250ce098c8e6-kube-api-access-tmt4v\") pod \"console-5d67c6549f-t9kz4\" (UID: \"af48efd8-d70a-4bc3-9f4f-250ce098c8e6\") " pod="openshift-console/console-5d67c6549f-t9kz4" Sep 10 15:42:58.285623 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:58.285516 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/af48efd8-d70a-4bc3-9f4f-250ce098c8e6-console-serving-cert\") pod \"console-5d67c6549f-t9kz4\" (UID: \"af48efd8-d70a-4bc3-9f4f-250ce098c8e6\") " pod="openshift-console/console-5d67c6549f-t9kz4" Sep 10 15:42:58.285623 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:58.285556 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/af48efd8-d70a-4bc3-9f4f-250ce098c8e6-console-config\") pod \"console-5d67c6549f-t9kz4\" (UID: \"af48efd8-d70a-4bc3-9f4f-250ce098c8e6\") " pod="openshift-console/console-5d67c6549f-t9kz4" Sep 10 15:42:58.285623 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:58.285578 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/af48efd8-d70a-4bc3-9f4f-250ce098c8e6-service-ca\") pod \"console-5d67c6549f-t9kz4\" (UID: \"af48efd8-d70a-4bc3-9f4f-250ce098c8e6\") " pod="openshift-console/console-5d67c6549f-t9kz4" Sep 10 15:42:58.285623 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:58.285608 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/af48efd8-d70a-4bc3-9f4f-250ce098c8e6-console-oauth-config\") pod \"console-5d67c6549f-t9kz4\" (UID: \"af48efd8-d70a-4bc3-9f4f-250ce098c8e6\") " pod="openshift-console/console-5d67c6549f-t9kz4" Sep 10 15:42:58.285857 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:58.285660 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/af48efd8-d70a-4bc3-9f4f-250ce098c8e6-oauth-serving-cert\") pod \"console-5d67c6549f-t9kz4\" (UID: \"af48efd8-d70a-4bc3-9f4f-250ce098c8e6\") " pod="openshift-console/console-5d67c6549f-t9kz4" Sep 10 15:42:58.285857 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:58.285710 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af48efd8-d70a-4bc3-9f4f-250ce098c8e6-trusted-ca-bundle\") pod \"console-5d67c6549f-t9kz4\" (UID: \"af48efd8-d70a-4bc3-9f4f-250ce098c8e6\") " pod="openshift-console/console-5d67c6549f-t9kz4" Sep 10 15:42:58.286916 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:58.286401 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/af48efd8-d70a-4bc3-9f4f-250ce098c8e6-console-config\") pod \"console-5d67c6549f-t9kz4\" (UID: \"af48efd8-d70a-4bc3-9f4f-250ce098c8e6\") " pod="openshift-console/console-5d67c6549f-t9kz4" Sep 10 15:42:58.286916 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:58.286875 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/af48efd8-d70a-4bc3-9f4f-250ce098c8e6-service-ca\") pod \"console-5d67c6549f-t9kz4\" (UID: \"af48efd8-d70a-4bc3-9f4f-250ce098c8e6\") " pod="openshift-console/console-5d67c6549f-t9kz4" Sep 10 15:42:58.287076 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:58.287019 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af48efd8-d70a-4bc3-9f4f-250ce098c8e6-trusted-ca-bundle\") pod \"console-5d67c6549f-t9kz4\" (UID: \"af48efd8-d70a-4bc3-9f4f-250ce098c8e6\") " pod="openshift-console/console-5d67c6549f-t9kz4" Sep 10 15:42:58.287284 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:58.287262 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/af48efd8-d70a-4bc3-9f4f-250ce098c8e6-oauth-serving-cert\") pod \"console-5d67c6549f-t9kz4\" (UID: \"af48efd8-d70a-4bc3-9f4f-250ce098c8e6\") " pod="openshift-console/console-5d67c6549f-t9kz4" Sep 10 15:42:58.289560 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:58.289526 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/af48efd8-d70a-4bc3-9f4f-250ce098c8e6-console-oauth-config\") pod \"console-5d67c6549f-t9kz4\" (UID: \"af48efd8-d70a-4bc3-9f4f-250ce098c8e6\") " pod="openshift-console/console-5d67c6549f-t9kz4" Sep 10 15:42:58.289830 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:58.289809 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/af48efd8-d70a-4bc3-9f4f-250ce098c8e6-console-serving-cert\") pod \"console-5d67c6549f-t9kz4\" (UID: \"af48efd8-d70a-4bc3-9f4f-250ce098c8e6\") " pod="openshift-console/console-5d67c6549f-t9kz4" Sep 10 15:42:58.295747 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:58.295710 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmt4v\" (UniqueName: \"kubernetes.io/projected/af48efd8-d70a-4bc3-9f4f-250ce098c8e6-kube-api-access-tmt4v\") pod \"console-5d67c6549f-t9kz4\" (UID: \"af48efd8-d70a-4bc3-9f4f-250ce098c8e6\") " pod="openshift-console/console-5d67c6549f-t9kz4" Sep 10 15:42:58.320644 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:58.320439 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d67c6549f-t9kz4" Sep 10 15:42:58.480995 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:58.480907 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wqh2d" event={"ID":"0d22a485-8c50-428d-8da7-8ca4e831c2c8","Type":"ContainerStarted","Data":"c67411b665e5ff6824219ca805cfc25750b6e55be30ef1426949c2efc8d9dd26"} Sep 10 15:42:59.302767 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:59.302711 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:59.425237 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:59.425207 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:59.696640 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:42:59.696546 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:00.225406 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:00.225317 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d67c6549f-t9kz4"] Sep 10 15:43:00.227978 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:43:00.227953 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf48efd8_d70a_4bc3_9f4f_250ce098c8e6.slice/crio-cbf9357a8da3e2663a1b3247e70728e94128599eaaf4f4a1d46ea43f02b22b79 WatchSource:0}: Error finding container cbf9357a8da3e2663a1b3247e70728e94128599eaaf4f4a1d46ea43f02b22b79: Status 404 returned error can't find the container with id cbf9357a8da3e2663a1b3247e70728e94128599eaaf4f4a1d46ea43f02b22b79 Sep 10 15:43:00.492013 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:00.491931 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d67c6549f-t9kz4" event={"ID":"af48efd8-d70a-4bc3-9f4f-250ce098c8e6","Type":"ContainerStarted","Data":"cbf9357a8da3e2663a1b3247e70728e94128599eaaf4f4a1d46ea43f02b22b79"} Sep 10 15:43:00.493435 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:00.493411 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8rhvj" event={"ID":"9191d5ea-e8a5-4674-86cd-05d2c6949141","Type":"ContainerStarted","Data":"7cf6f6745af71bb9a6f25f1ec6957beafea116606aee68389de54e89f402dd32"} Sep 10 15:43:00.495550 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:00.495519 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wqh2d" event={"ID":"0d22a485-8c50-428d-8da7-8ca4e831c2c8","Type":"ContainerStarted","Data":"6eeb9f25ccda9ca16eec5200841518a7e0b36c08b536aff40d56991955172c43"} Sep 10 15:43:00.497355 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:00.497334 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nt5ps" event={"ID":"93cdfcd5-7e56-4acf-ac1b-eb9896ba6581","Type":"ContainerStarted","Data":"f8d2bd7eee59c88438928b3995610fecfba1849bffbfb00143fca387c69778f3"} Sep 10 15:43:00.497454 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:00.497378 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nt5ps" event={"ID":"93cdfcd5-7e56-4acf-ac1b-eb9896ba6581","Type":"ContainerStarted","Data":"a8fb0ce1a8d09fa9a22613d5c4cc78d78f8b93d7dfc476086e1f91e35a4e100c"} Sep 10 15:43:00.497504 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:00.497486 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-nt5ps" Sep 10 15:43:00.511360 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:00.511239 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8rhvj" podStartSLOduration=1.696265091 podStartE2EDuration="4.511224768s" totalImagesPullingTime="2.814959677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:42:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:43:00.511127809 +0000 UTC m=+57.864646989" watchObservedRunningTime="2026-09-10 15:43:00.511224768 +0000 UTC m=+57.864743931" Sep 10 15:43:00.531140 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:00.531096 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-nt5ps" podStartSLOduration=1.694931869 podStartE2EDuration="4.531080101s" totalImagesPullingTime="2.836148232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:42:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:43:00.53107995 +0000 UTC m=+57.884599132" watchObservedRunningTime="2026-09-10 15:43:00.531080101 +0000 UTC m=+57.884599263" Sep 10 15:43:00.552372 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:00.552328 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-wqh2d" podStartSLOduration=1.8676683010000001 podStartE2EDuration="4.55231248s" totalImagesPullingTime="2.684644179s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:42:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:43:00.552122957 +0000 UTC m=+57.905642142" watchObservedRunningTime="2026-09-10 15:43:00.55231248 +0000 UTC m=+57.905831641" Sep 10 15:43:02.270533 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:02.270488 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:03.509769 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:03.509716 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d67c6549f-t9kz4" event={"ID":"af48efd8-d70a-4bc3-9f4f-250ce098c8e6","Type":"ContainerStarted","Data":"52cb12023a69aaa97215810660cfd699027acfbb9910bdeb7db1629238e38caa"} Sep 10 15:43:03.527376 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:03.527331 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5d67c6549f-t9kz4" podStartSLOduration=3.77554381 podStartE2EDuration="6.527317094s" totalImagesPullingTime="2.751773284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:42:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:43:03.526683593 +0000 UTC m=+60.880202755" watchObservedRunningTime="2026-09-10 15:43:03.527317094 +0000 UTC m=+60.880836256" Sep 10 15:43:04.404523 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.404484 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:04.429387 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.429364 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zc9b9" Sep 10 15:43:04.502446 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.502407 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:04.607451 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.607414 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:04.655975 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.655913 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-tz8fc"] Sep 10 15:43:04.687508 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.687484 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-bw9q7"] Sep 10 15:43:04.687649 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.687619 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-tz8fc" Sep 10 15:43:04.690223 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.690187 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 10 15:43:04.690379 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.690239 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 10 15:43:04.691051 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.690948 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Sep 10 15:43:04.691180 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.690906 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-xqd94\"" Sep 10 15:43:04.691955 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.691826 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Sep 10 15:43:04.692959 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.692929 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 10 15:43:04.701436 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.701413 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:04.705574 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.705556 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-tz8fc"] Sep 10 15:43:04.705628 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.705596 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-bw9q7"] Sep 10 15:43:04.705676 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.705635 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-k8jsb"] Sep 10 15:43:04.705798 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.705785 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-bw9q7" Sep 10 15:43:04.708130 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.708102 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 10 15:43:04.708217 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.708173 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 10 15:43:04.708383 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.708369 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 10 15:43:04.708581 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.708561 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-7df9j\"" Sep 10 15:43:04.722518 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.722498 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-k8jsb" Sep 10 15:43:04.724869 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.724849 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 10 15:43:04.725085 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.725066 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 10 15:43:04.725356 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.725340 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-j55xj\"" Sep 10 15:43:04.725444 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.725429 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 10 15:43:04.732307 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.732290 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d78d8162-f3d9-4730-8354-e8eb6d8e676c-sys\") pod \"node-exporter-k8jsb\" (UID: \"d78d8162-f3d9-4730-8354-e8eb6d8e676c\") " pod="openshift-monitoring/node-exporter-k8jsb" Sep 10 15:43:04.732383 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.732314 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmcqs\" (UniqueName: \"kubernetes.io/projected/d78d8162-f3d9-4730-8354-e8eb6d8e676c-kube-api-access-bmcqs\") pod \"node-exporter-k8jsb\" (UID: \"d78d8162-f3d9-4730-8354-e8eb6d8e676c\") " pod="openshift-monitoring/node-exporter-k8jsb" Sep 10 15:43:04.732383 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.732333 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d78d8162-f3d9-4730-8354-e8eb6d8e676c-node-exporter-textfile\") pod \"node-exporter-k8jsb\" (UID: \"d78d8162-f3d9-4730-8354-e8eb6d8e676c\") " pod="openshift-monitoring/node-exporter-k8jsb" Sep 10 15:43:04.732383 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.732351 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j86dz\" (UniqueName: \"kubernetes.io/projected/566fd587-ad31-49c7-8480-62ace5e8c065-kube-api-access-j86dz\") pod \"openshift-state-metrics-657fd6c54-tz8fc\" (UID: \"566fd587-ad31-49c7-8480-62ace5e8c065\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-tz8fc" Sep 10 15:43:04.732550 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.732411 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0a661843-e27a-4388-b301-49c9eafddf18-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-bw9q7\" (UID: \"0a661843-e27a-4388-b301-49c9eafddf18\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bw9q7" Sep 10 15:43:04.732550 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.732467 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/566fd587-ad31-49c7-8480-62ace5e8c065-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-tz8fc\" (UID: \"566fd587-ad31-49c7-8480-62ace5e8c065\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-tz8fc" Sep 10 15:43:04.732550 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.732500 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0a661843-e27a-4388-b301-49c9eafddf18-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-bw9q7\" (UID: \"0a661843-e27a-4388-b301-49c9eafddf18\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bw9q7" Sep 10 15:43:04.732550 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.732533 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/566fd587-ad31-49c7-8480-62ace5e8c065-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-tz8fc\" (UID: \"566fd587-ad31-49c7-8480-62ace5e8c065\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-tz8fc" Sep 10 15:43:04.732756 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.732570 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/566fd587-ad31-49c7-8480-62ace5e8c065-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-tz8fc\" (UID: \"566fd587-ad31-49c7-8480-62ace5e8c065\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-tz8fc" Sep 10 15:43:04.732756 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.732599 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0a661843-e27a-4388-b301-49c9eafddf18-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-bw9q7\" (UID: \"0a661843-e27a-4388-b301-49c9eafddf18\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bw9q7" Sep 10 15:43:04.732756 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.732627 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d78d8162-f3d9-4730-8354-e8eb6d8e676c-metrics-client-ca\") pod \"node-exporter-k8jsb\" (UID: \"d78d8162-f3d9-4730-8354-e8eb6d8e676c\") " pod="openshift-monitoring/node-exporter-k8jsb" Sep 10 15:43:04.732756 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.732674 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d78d8162-f3d9-4730-8354-e8eb6d8e676c-root\") pod \"node-exporter-k8jsb\" (UID: \"d78d8162-f3d9-4730-8354-e8eb6d8e676c\") " pod="openshift-monitoring/node-exporter-k8jsb" Sep 10 15:43:04.732756 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.732703 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a661843-e27a-4388-b301-49c9eafddf18-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-bw9q7\" (UID: \"0a661843-e27a-4388-b301-49c9eafddf18\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bw9q7" Sep 10 15:43:04.732920 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.732780 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0a661843-e27a-4388-b301-49c9eafddf18-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-bw9q7\" (UID: \"0a661843-e27a-4388-b301-49c9eafddf18\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bw9q7" Sep 10 15:43:04.732920 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.732842 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d78d8162-f3d9-4730-8354-e8eb6d8e676c-node-exporter-wtmp\") pod \"node-exporter-k8jsb\" (UID: \"d78d8162-f3d9-4730-8354-e8eb6d8e676c\") " pod="openshift-monitoring/node-exporter-k8jsb" Sep 10 15:43:04.732920 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.732866 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcf7b\" (UniqueName: \"kubernetes.io/projected/0a661843-e27a-4388-b301-49c9eafddf18-kube-api-access-pcf7b\") pod \"kube-state-metrics-74cb57df7-bw9q7\" (UID: \"0a661843-e27a-4388-b301-49c9eafddf18\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bw9q7" Sep 10 15:43:04.732920 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.732896 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d78d8162-f3d9-4730-8354-e8eb6d8e676c-node-exporter-accelerators-collector-config\") pod \"node-exporter-k8jsb\" (UID: \"d78d8162-f3d9-4730-8354-e8eb6d8e676c\") " pod="openshift-monitoring/node-exporter-k8jsb" Sep 10 15:43:04.733054 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.732968 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d78d8162-f3d9-4730-8354-e8eb6d8e676c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-k8jsb\" (UID: \"d78d8162-f3d9-4730-8354-e8eb6d8e676c\") " pod="openshift-monitoring/node-exporter-k8jsb" Sep 10 15:43:04.733054 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.732995 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d78d8162-f3d9-4730-8354-e8eb6d8e676c-node-exporter-tls\") pod \"node-exporter-k8jsb\" (UID: \"d78d8162-f3d9-4730-8354-e8eb6d8e676c\") " pod="openshift-monitoring/node-exporter-k8jsb" Sep 10 15:43:04.792008 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.791975 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:04.833980 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.833956 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d78d8162-f3d9-4730-8354-e8eb6d8e676c-metrics-client-ca\") pod \"node-exporter-k8jsb\" (UID: \"d78d8162-f3d9-4730-8354-e8eb6d8e676c\") " pod="openshift-monitoring/node-exporter-k8jsb" Sep 10 15:43:04.834057 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.833987 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d78d8162-f3d9-4730-8354-e8eb6d8e676c-root\") pod \"node-exporter-k8jsb\" (UID: \"d78d8162-f3d9-4730-8354-e8eb6d8e676c\") " pod="openshift-monitoring/node-exporter-k8jsb" Sep 10 15:43:04.834057 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.834005 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a661843-e27a-4388-b301-49c9eafddf18-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-bw9q7\" (UID: \"0a661843-e27a-4388-b301-49c9eafddf18\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bw9q7" Sep 10 15:43:04.834057 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.834023 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0a661843-e27a-4388-b301-49c9eafddf18-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-bw9q7\" (UID: \"0a661843-e27a-4388-b301-49c9eafddf18\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bw9q7" Sep 10 15:43:04.834228 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.834056 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d78d8162-f3d9-4730-8354-e8eb6d8e676c-node-exporter-wtmp\") pod \"node-exporter-k8jsb\" (UID: \"d78d8162-f3d9-4730-8354-e8eb6d8e676c\") " pod="openshift-monitoring/node-exporter-k8jsb" Sep 10 15:43:04.834228 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.834079 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcf7b\" (UniqueName: \"kubernetes.io/projected/0a661843-e27a-4388-b301-49c9eafddf18-kube-api-access-pcf7b\") pod \"kube-state-metrics-74cb57df7-bw9q7\" (UID: \"0a661843-e27a-4388-b301-49c9eafddf18\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bw9q7" Sep 10 15:43:04.834228 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.834096 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d78d8162-f3d9-4730-8354-e8eb6d8e676c-root\") pod \"node-exporter-k8jsb\" (UID: \"d78d8162-f3d9-4730-8354-e8eb6d8e676c\") " pod="openshift-monitoring/node-exporter-k8jsb" Sep 10 15:43:04.834228 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.834108 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d78d8162-f3d9-4730-8354-e8eb6d8e676c-node-exporter-accelerators-collector-config\") pod \"node-exporter-k8jsb\" (UID: \"d78d8162-f3d9-4730-8354-e8eb6d8e676c\") " pod="openshift-monitoring/node-exporter-k8jsb" Sep 10 15:43:04.834228 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:43:04.834164 2576 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Sep 10 15:43:04.834228 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.834173 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d78d8162-f3d9-4730-8354-e8eb6d8e676c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-k8jsb\" (UID: \"d78d8162-f3d9-4730-8354-e8eb6d8e676c\") " pod="openshift-monitoring/node-exporter-k8jsb" Sep 10 15:43:04.834228 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.834207 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d78d8162-f3d9-4730-8354-e8eb6d8e676c-node-exporter-tls\") pod \"node-exporter-k8jsb\" (UID: \"d78d8162-f3d9-4730-8354-e8eb6d8e676c\") " pod="openshift-monitoring/node-exporter-k8jsb" Sep 10 15:43:04.834558 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:43:04.834238 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a661843-e27a-4388-b301-49c9eafddf18-kube-state-metrics-tls podName:0a661843-e27a-4388-b301-49c9eafddf18 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:05.334218339 +0000 UTC m=+62.687737479 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/0a661843-e27a-4388-b301-49c9eafddf18-kube-state-metrics-tls") pod "kube-state-metrics-74cb57df7-bw9q7" (UID: "0a661843-e27a-4388-b301-49c9eafddf18") : secret "kube-state-metrics-tls" not found Sep 10 15:43:04.834558 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.834286 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d78d8162-f3d9-4730-8354-e8eb6d8e676c-sys\") pod \"node-exporter-k8jsb\" (UID: \"d78d8162-f3d9-4730-8354-e8eb6d8e676c\") " pod="openshift-monitoring/node-exporter-k8jsb" Sep 10 15:43:04.834558 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.834321 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmcqs\" (UniqueName: \"kubernetes.io/projected/d78d8162-f3d9-4730-8354-e8eb6d8e676c-kube-api-access-bmcqs\") pod \"node-exporter-k8jsb\" (UID: \"d78d8162-f3d9-4730-8354-e8eb6d8e676c\") " pod="openshift-monitoring/node-exporter-k8jsb" Sep 10 15:43:04.834558 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.834350 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d78d8162-f3d9-4730-8354-e8eb6d8e676c-node-exporter-textfile\") pod \"node-exporter-k8jsb\" (UID: \"d78d8162-f3d9-4730-8354-e8eb6d8e676c\") " pod="openshift-monitoring/node-exporter-k8jsb" Sep 10 15:43:04.834558 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.834375 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j86dz\" (UniqueName: \"kubernetes.io/projected/566fd587-ad31-49c7-8480-62ace5e8c065-kube-api-access-j86dz\") pod \"openshift-state-metrics-657fd6c54-tz8fc\" (UID: \"566fd587-ad31-49c7-8480-62ace5e8c065\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-tz8fc" Sep 10 15:43:04.834558 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.834403 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0a661843-e27a-4388-b301-49c9eafddf18-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-bw9q7\" (UID: \"0a661843-e27a-4388-b301-49c9eafddf18\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bw9q7" Sep 10 15:43:04.834558 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.834410 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d78d8162-f3d9-4730-8354-e8eb6d8e676c-sys\") pod \"node-exporter-k8jsb\" (UID: \"d78d8162-f3d9-4730-8354-e8eb6d8e676c\") " pod="openshift-monitoring/node-exporter-k8jsb" Sep 10 15:43:04.834558 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.834236 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d78d8162-f3d9-4730-8354-e8eb6d8e676c-node-exporter-wtmp\") pod \"node-exporter-k8jsb\" (UID: \"d78d8162-f3d9-4730-8354-e8eb6d8e676c\") " pod="openshift-monitoring/node-exporter-k8jsb" Sep 10 15:43:04.834558 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.834455 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/566fd587-ad31-49c7-8480-62ace5e8c065-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-tz8fc\" (UID: \"566fd587-ad31-49c7-8480-62ace5e8c065\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-tz8fc" Sep 10 15:43:04.834558 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.834486 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0a661843-e27a-4388-b301-49c9eafddf18-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-bw9q7\" (UID: \"0a661843-e27a-4388-b301-49c9eafddf18\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bw9q7" Sep 10 15:43:04.834558 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.834524 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/566fd587-ad31-49c7-8480-62ace5e8c065-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-tz8fc\" (UID: \"566fd587-ad31-49c7-8480-62ace5e8c065\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-tz8fc" Sep 10 15:43:04.835119 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.834566 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/566fd587-ad31-49c7-8480-62ace5e8c065-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-tz8fc\" (UID: \"566fd587-ad31-49c7-8480-62ace5e8c065\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-tz8fc" Sep 10 15:43:04.835119 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.834589 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0a661843-e27a-4388-b301-49c9eafddf18-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-bw9q7\" (UID: \"0a661843-e27a-4388-b301-49c9eafddf18\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bw9q7" Sep 10 15:43:04.835119 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.834690 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d78d8162-f3d9-4730-8354-e8eb6d8e676c-metrics-client-ca\") pod \"node-exporter-k8jsb\" (UID: \"d78d8162-f3d9-4730-8354-e8eb6d8e676c\") " pod="openshift-monitoring/node-exporter-k8jsb" Sep 10 15:43:04.835119 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.834721 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d78d8162-f3d9-4730-8354-e8eb6d8e676c-node-exporter-accelerators-collector-config\") pod \"node-exporter-k8jsb\" (UID: \"d78d8162-f3d9-4730-8354-e8eb6d8e676c\") " pod="openshift-monitoring/node-exporter-k8jsb" Sep 10 15:43:04.835119 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.834747 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d78d8162-f3d9-4730-8354-e8eb6d8e676c-node-exporter-textfile\") pod \"node-exporter-k8jsb\" (UID: \"d78d8162-f3d9-4730-8354-e8eb6d8e676c\") " pod="openshift-monitoring/node-exporter-k8jsb" Sep 10 15:43:04.835119 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.834803 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a661843-e27a-4388-b301-49c9eafddf18-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-bw9q7\" (UID: \"0a661843-e27a-4388-b301-49c9eafddf18\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bw9q7" Sep 10 15:43:04.835119 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.834922 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0a661843-e27a-4388-b301-49c9eafddf18-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-bw9q7\" (UID: \"0a661843-e27a-4388-b301-49c9eafddf18\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bw9q7" Sep 10 15:43:04.835633 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.835370 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0a661843-e27a-4388-b301-49c9eafddf18-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-bw9q7\" (UID: \"0a661843-e27a-4388-b301-49c9eafddf18\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bw9q7" Sep 10 15:43:04.835687 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.835649 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/566fd587-ad31-49c7-8480-62ace5e8c065-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-tz8fc\" (UID: \"566fd587-ad31-49c7-8480-62ace5e8c065\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-tz8fc" Sep 10 15:43:04.837262 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.837242 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d78d8162-f3d9-4730-8354-e8eb6d8e676c-node-exporter-tls\") pod \"node-exporter-k8jsb\" (UID: \"d78d8162-f3d9-4730-8354-e8eb6d8e676c\") " pod="openshift-monitoring/node-exporter-k8jsb" Sep 10 15:43:04.837414 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.837391 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/566fd587-ad31-49c7-8480-62ace5e8c065-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-tz8fc\" (UID: \"566fd587-ad31-49c7-8480-62ace5e8c065\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-tz8fc" Sep 10 15:43:04.837414 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.837402 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0a661843-e27a-4388-b301-49c9eafddf18-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-bw9q7\" (UID: \"0a661843-e27a-4388-b301-49c9eafddf18\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bw9q7" Sep 10 15:43:04.837820 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.837801 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/566fd587-ad31-49c7-8480-62ace5e8c065-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-tz8fc\" (UID: \"566fd587-ad31-49c7-8480-62ace5e8c065\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-tz8fc" Sep 10 15:43:04.838055 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.838036 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d78d8162-f3d9-4730-8354-e8eb6d8e676c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-k8jsb\" (UID: \"d78d8162-f3d9-4730-8354-e8eb6d8e676c\") " pod="openshift-monitoring/node-exporter-k8jsb" Sep 10 15:43:04.845874 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.845848 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j86dz\" (UniqueName: \"kubernetes.io/projected/566fd587-ad31-49c7-8480-62ace5e8c065-kube-api-access-j86dz\") pod \"openshift-state-metrics-657fd6c54-tz8fc\" (UID: \"566fd587-ad31-49c7-8480-62ace5e8c065\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-tz8fc" Sep 10 15:43:04.846180 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.846160 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmcqs\" (UniqueName: \"kubernetes.io/projected/d78d8162-f3d9-4730-8354-e8eb6d8e676c-kube-api-access-bmcqs\") pod \"node-exporter-k8jsb\" (UID: \"d78d8162-f3d9-4730-8354-e8eb6d8e676c\") " pod="openshift-monitoring/node-exporter-k8jsb" Sep 10 15:43:04.846415 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.846394 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcf7b\" (UniqueName: \"kubernetes.io/projected/0a661843-e27a-4388-b301-49c9eafddf18-kube-api-access-pcf7b\") pod \"kube-state-metrics-74cb57df7-bw9q7\" (UID: \"0a661843-e27a-4388-b301-49c9eafddf18\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bw9q7" Sep 10 15:43:04.893271 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.893238 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:04.994249 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.994148 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:04.999934 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:04.999905 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-tz8fc" Sep 10 15:43:05.030250 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.030224 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-k8jsb" Sep 10 15:43:05.040448 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:43:05.040422 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd78d8162_f3d9_4730_8354_e8eb6d8e676c.slice/crio-144fc891825f2fa4c4becfb8ce1e1dfbac9330b8d2340346cde8165b3df2550a WatchSource:0}: Error finding container 144fc891825f2fa4c4becfb8ce1e1dfbac9330b8d2340346cde8165b3df2550a: Status 404 returned error can't find the container with id 144fc891825f2fa4c4becfb8ce1e1dfbac9330b8d2340346cde8165b3df2550a Sep 10 15:43:05.093454 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.093424 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:05.194220 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.194183 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:05.273508 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.273424 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-tz8fc"] Sep 10 15:43:05.276167 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:43:05.276141 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod566fd587_ad31_49c7_8480_62ace5e8c065.slice/crio-34680d8106d156567b41ff9842007b7df11a8e7848ef76b835ab8cb0508ebd90 WatchSource:0}: Error finding container 34680d8106d156567b41ff9842007b7df11a8e7848ef76b835ab8cb0508ebd90: Status 404 returned error can't find the container with id 34680d8106d156567b41ff9842007b7df11a8e7848ef76b835ab8cb0508ebd90 Sep 10 15:43:05.339816 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.339788 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0a661843-e27a-4388-b301-49c9eafddf18-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-bw9q7\" (UID: \"0a661843-e27a-4388-b301-49c9eafddf18\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bw9q7" Sep 10 15:43:05.342500 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.342465 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0a661843-e27a-4388-b301-49c9eafddf18-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-bw9q7\" (UID: \"0a661843-e27a-4388-b301-49c9eafddf18\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bw9q7" Sep 10 15:43:05.517514 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.517481 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k8jsb" event={"ID":"d78d8162-f3d9-4730-8354-e8eb6d8e676c","Type":"ContainerStarted","Data":"144fc891825f2fa4c4becfb8ce1e1dfbac9330b8d2340346cde8165b3df2550a"} Sep 10 15:43:05.519336 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.519302 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-tz8fc" event={"ID":"566fd587-ad31-49c7-8480-62ace5e8c065","Type":"ContainerStarted","Data":"c800f55767565e4e0b666ea22766e8c6fd8aa9c84454de08c4baa118ed40a697"} Sep 10 15:43:05.519435 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.519347 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-tz8fc" event={"ID":"566fd587-ad31-49c7-8480-62ace5e8c065","Type":"ContainerStarted","Data":"c457961e9d6e244b4e88738673852d7cb3be7d8573d221bb0fcb63b3165fc229"} Sep 10 15:43:05.519435 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.519364 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-tz8fc" event={"ID":"566fd587-ad31-49c7-8480-62ace5e8c065","Type":"ContainerStarted","Data":"34680d8106d156567b41ff9842007b7df11a8e7848ef76b835ab8cb0508ebd90"} Sep 10 15:43:05.614644 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.614611 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-bw9q7" Sep 10 15:43:05.629383 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.629354 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:05.711706 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.711637 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:05.715769 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.715745 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 15:43:05.719646 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.719623 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:05.735452 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.735176 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 10 15:43:05.735452 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.735255 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 10 15:43:05.735452 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.735176 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 10 15:43:05.735452 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.735315 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 10 15:43:05.735452 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.735260 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 10 15:43:05.735452 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.735261 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 10 15:43:05.735864 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.735644 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 10 15:43:05.735864 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.735804 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-96vnm\"" Sep 10 15:43:05.735961 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.735807 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 10 15:43:05.736654 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.736637 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 10 15:43:05.742768 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.742716 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 15:43:05.743008 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.742981 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:05.743133 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.743035 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-config-volume\") pod \"alertmanager-main-0\" (UID: \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:05.743133 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.743060 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-web-config\") pod \"alertmanager-main-0\" (UID: \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:05.743133 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.743091 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:05.743133 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.743106 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:05.743133 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.743128 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:05.743366 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.743153 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:05.743366 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.743168 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:05.743366 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.743218 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:05.743366 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.743254 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvp7v\" (UniqueName: \"kubernetes.io/projected/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-kube-api-access-rvp7v\") pod \"alertmanager-main-0\" (UID: \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:05.743366 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.743277 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-config-out\") pod \"alertmanager-main-0\" (UID: \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:05.743366 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.743323 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:05.743366 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.743347 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:05.843906 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.843869 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-config-volume\") pod \"alertmanager-main-0\" (UID: \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:05.843906 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.843905 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-web-config\") pod \"alertmanager-main-0\" (UID: \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:05.844139 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.843923 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:05.844139 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.843944 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:05.844139 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.843969 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:05.844139 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.844096 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:05.844331 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.844140 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:05.844331 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.844188 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:05.844331 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.844238 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvp7v\" (UniqueName: \"kubernetes.io/projected/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-kube-api-access-rvp7v\") pod \"alertmanager-main-0\" (UID: \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:05.844331 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.844273 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-config-out\") pod \"alertmanager-main-0\" (UID: \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:05.844331 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.844305 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:05.844558 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.844340 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:05.844558 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:43:05.844498 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-alertmanager-trusted-ca-bundle podName:d9dd9c02-cf57-49d8-a55c-0139a15c7eeb nodeName:}" failed. No retries permitted until 2026-09-10 15:43:06.344476055 +0000 UTC m=+63.697995195 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "d9dd9c02-cf57-49d8-a55c-0139a15c7eeb") : configmap references non-existent config key: ca-bundle.crt Sep 10 15:43:05.844873 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.844848 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:05.845099 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.845078 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:05.845990 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.845937 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:05.847272 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.847250 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-config-out\") pod \"alertmanager-main-0\" (UID: \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:05.848090 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.848066 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:05.848503 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.848480 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:05.848895 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.848877 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-config-volume\") pod \"alertmanager-main-0\" (UID: \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:05.848895 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.848903 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:05.849182 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.849165 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-web-config\") pod \"alertmanager-main-0\" (UID: \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:05.849489 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.849472 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:05.849864 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.849844 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:05.851078 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.851056 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:05.865559 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.865423 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvp7v\" (UniqueName: \"kubernetes.io/projected/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-kube-api-access-rvp7v\") pod \"alertmanager-main-0\" (UID: \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:05.906619 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.906586 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-bw9q7"] Sep 10 15:43:05.910067 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:43:05.910039 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a661843_e27a_4388_b301_49c9eafddf18.slice/crio-2df63c5dd71694e4e41a11d45d11136ea631bfcaba4107e66f406b63208e90f2 WatchSource:0}: Error finding container 2df63c5dd71694e4e41a11d45d11136ea631bfcaba4107e66f406b63208e90f2: Status 404 returned error can't find the container with id 2df63c5dd71694e4e41a11d45d11136ea631bfcaba4107e66f406b63208e90f2 Sep 10 15:43:05.927869 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:05.927399 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:06.327057 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:06.327012 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:06.350313 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:06.350273 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:06.351263 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:06.351237 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:06.396926 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:06.396823 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:06.529582 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:06.529541 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-bw9q7" event={"ID":"0a661843-e27a-4388-b301-49c9eafddf18","Type":"ContainerStarted","Data":"2df63c5dd71694e4e41a11d45d11136ea631bfcaba4107e66f406b63208e90f2"} Sep 10 15:43:06.629839 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:06.629812 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:06.914718 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:06.914643 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 15:43:06.918551 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:43:06.918516 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9dd9c02_cf57_49d8_a55c_0139a15c7eeb.slice/crio-f34a91977ceef34c5aace658054d9bccffe1bd86bc0ba402165193a583f08523 WatchSource:0}: Error finding container f34a91977ceef34c5aace658054d9bccffe1bd86bc0ba402165193a583f08523: Status 404 returned error can't find the container with id f34a91977ceef34c5aace658054d9bccffe1bd86bc0ba402165193a583f08523 Sep 10 15:43:07.534704 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:07.534674 2576 generic.go:356] "Generic (PLEG): container finished" podID="d78d8162-f3d9-4730-8354-e8eb6d8e676c" containerID="2d165974531ab7c52a648872778659bb946e77e9935aa4dc5103bb70c21fc541" exitCode=0 Sep 10 15:43:07.534863 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:07.534778 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k8jsb" event={"ID":"d78d8162-f3d9-4730-8354-e8eb6d8e676c","Type":"ContainerDied","Data":"2d165974531ab7c52a648872778659bb946e77e9935aa4dc5103bb70c21fc541"} Sep 10 15:43:07.537709 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:07.537510 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-tz8fc" event={"ID":"566fd587-ad31-49c7-8480-62ace5e8c065","Type":"ContainerStarted","Data":"eaf3414808f5dbba3663883924f647372f5688632ff7b0577917d5583b8e7c14"} Sep 10 15:43:07.539708 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:07.539684 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-bw9q7" event={"ID":"0a661843-e27a-4388-b301-49c9eafddf18","Type":"ContainerStarted","Data":"4286c5992fe51688728a2df3c3789cf9a55667152fd1ed77dff164a14091a41e"} Sep 10 15:43:07.539846 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:07.539762 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-bw9q7" event={"ID":"0a661843-e27a-4388-b301-49c9eafddf18","Type":"ContainerStarted","Data":"323751731cada7f67edd904d731a703f66224a6a14226e546a9e03db23d8cf6d"} Sep 10 15:43:07.541355 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:07.541336 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb","Type":"ContainerStarted","Data":"f34a91977ceef34c5aace658054d9bccffe1bd86bc0ba402165193a583f08523"} Sep 10 15:43:07.653019 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:07.652963 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-tz8fc" podStartSLOduration=2.560685144 podStartE2EDuration="3.652940864s" totalImagesPullingTime="1.09225572s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:43:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:43:07.652150866 +0000 UTC m=+65.005670045" watchObservedRunningTime="2026-09-10 15:43:07.652940864 +0000 UTC m=+65.006460025" Sep 10 15:43:07.781980 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:07.781942 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:07.861398 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:07.861359 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1b1f777f-8ed4-408b-9721-80c9ae47af6e-original-pull-secret\") pod \"global-pull-secret-syncer-hzjqc\" (UID: \"1b1f777f-8ed4-408b-9721-80c9ae47af6e\") " pod="kube-system/global-pull-secret-syncer-hzjqc" Sep 10 15:43:07.864141 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:07.864118 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 10 15:43:07.870854 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:07.870824 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:07.874616 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:07.874587 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1b1f777f-8ed4-408b-9721-80c9ae47af6e-original-pull-secret\") pod \"global-pull-secret-syncer-hzjqc\" (UID: \"1b1f777f-8ed4-408b-9721-80c9ae47af6e\") " pod="kube-system/global-pull-secret-syncer-hzjqc" Sep 10 15:43:07.935665 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:07.935549 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:07.962862 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:07.962825 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppv8k\" (UniqueName: \"kubernetes.io/projected/c27ce7b1-1b71-4f92-8897-fd4748ebd262-kube-api-access-ppv8k\") pod \"network-check-target-ptjhb\" (UID: \"c27ce7b1-1b71-4f92-8897-fd4748ebd262\") " pod="openshift-network-diagnostics/network-check-target-ptjhb" Sep 10 15:43:07.962994 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:07.962911 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbfbb1ad-2b61-4739-926f-01774a102c98-metrics-certs\") pod \"network-metrics-daemon-9d4jc\" (UID: \"bbfbb1ad-2b61-4739-926f-01774a102c98\") " pod="openshift-multus/network-metrics-daemon-9d4jc" Sep 10 15:43:07.965594 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:07.965575 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 10 15:43:07.965681 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:07.965613 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 10 15:43:07.976465 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:07.976414 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbfbb1ad-2b61-4739-926f-01774a102c98-metrics-certs\") pod \"network-metrics-daemon-9d4jc\" (UID: \"bbfbb1ad-2b61-4739-926f-01774a102c98\") " pod="openshift-multus/network-metrics-daemon-9d4jc" Sep 10 15:43:07.976465 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:07.976435 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 10 15:43:07.978619 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:07.978602 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hzjqc" Sep 10 15:43:07.987293 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:07.987274 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppv8k\" (UniqueName: \"kubernetes.io/projected/c27ce7b1-1b71-4f92-8897-fd4748ebd262-kube-api-access-ppv8k\") pod \"network-check-target-ptjhb\" (UID: \"c27ce7b1-1b71-4f92-8897-fd4748ebd262\") " pod="openshift-network-diagnostics/network-check-target-ptjhb" Sep 10 15:43:08.078246 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:08.078213 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sjjtv\"" Sep 10 15:43:08.086607 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:08.086583 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ptjhb" Sep 10 15:43:08.192152 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:08.192098 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:08.276470 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:08.276444 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2gdj6\"" Sep 10 15:43:08.285194 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:08.285174 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9d4jc" Sep 10 15:43:08.306179 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:08.306156 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:08.321038 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:08.321016 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5d67c6549f-t9kz4" Sep 10 15:43:08.321136 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:08.321083 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d67c6549f-t9kz4" Sep 10 15:43:08.329194 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:08.329172 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5d67c6549f-t9kz4" Sep 10 15:43:08.377573 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:08.377551 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hzjqc"] Sep 10 15:43:08.378455 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:08.378400 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ptjhb"] Sep 10 15:43:08.380392 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:43:08.380369 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc27ce7b1_1b71_4f92_8897_fd4748ebd262.slice/crio-7daaed0d45991ad5d2f038f4142a674dfa0b81b4a96af18a31fea8e30a685e2f WatchSource:0}: Error finding container 7daaed0d45991ad5d2f038f4142a674dfa0b81b4a96af18a31fea8e30a685e2f: Status 404 returned error can't find the container with id 7daaed0d45991ad5d2f038f4142a674dfa0b81b4a96af18a31fea8e30a685e2f Sep 10 15:43:08.380922 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:43:08.380903 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b1f777f_8ed4_408b_9721_80c9ae47af6e.slice/crio-a21f261fd1ae231f0c1601ec71f126951a546238e89e67219ffe00a407b74e33 WatchSource:0}: Error finding container a21f261fd1ae231f0c1601ec71f126951a546238e89e67219ffe00a407b74e33: Status 404 returned error can't find the container with id a21f261fd1ae231f0c1601ec71f126951a546238e89e67219ffe00a407b74e33 Sep 10 15:43:08.401156 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:08.401133 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:08.465973 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:08.465899 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:08.547165 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:08.547133 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k8jsb" event={"ID":"d78d8162-f3d9-4730-8354-e8eb6d8e676c","Type":"ContainerStarted","Data":"d10bae54488216656b899cecab8d480a80d61b801eddce5ecbfa7f9948b1d687"} Sep 10 15:43:08.547330 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:08.547188 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k8jsb" event={"ID":"d78d8162-f3d9-4730-8354-e8eb6d8e676c","Type":"ContainerStarted","Data":"7acd7f7c9713f09ba2bc9cca668b7784d2803ad8da555500acb90be4cd2977f4"} Sep 10 15:43:08.549177 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:08.549153 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-bw9q7" event={"ID":"0a661843-e27a-4388-b301-49c9eafddf18","Type":"ContainerStarted","Data":"1cc740e95593cb27fe5b61e8311c3310176467218dc999c667a477204b7383ad"} Sep 10 15:43:08.550683 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:08.550664 2576 generic.go:356] "Generic (PLEG): container finished" podID="d9dd9c02-cf57-49d8-a55c-0139a15c7eeb" containerID="2bebb7dcd3bb8af8304a2d5b718d5a84a065932fddb60ce095460bb246dce756" exitCode=0 Sep 10 15:43:08.550805 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:08.550724 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb","Type":"ContainerDied","Data":"2bebb7dcd3bb8af8304a2d5b718d5a84a065932fddb60ce095460bb246dce756"} Sep 10 15:43:08.552080 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:08.552055 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hzjqc" event={"ID":"1b1f777f-8ed4-408b-9721-80c9ae47af6e","Type":"ContainerStarted","Data":"a21f261fd1ae231f0c1601ec71f126951a546238e89e67219ffe00a407b74e33"} Sep 10 15:43:08.553198 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:08.553177 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ptjhb" event={"ID":"c27ce7b1-1b71-4f92-8897-fd4748ebd262","Type":"ContainerStarted","Data":"7daaed0d45991ad5d2f038f4142a674dfa0b81b4a96af18a31fea8e30a685e2f"} Sep 10 15:43:08.557626 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:08.557585 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5d67c6549f-t9kz4" Sep 10 15:43:08.557904 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:08.557885 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9d4jc"] Sep 10 15:43:08.561640 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:43:08.561621 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbfbb1ad_2b61_4739_926f_01774a102c98.slice/crio-0347707d85d9423e7d181120cfe687a519e2721f5ae92532d323533780241b0c WatchSource:0}: Error finding container 0347707d85d9423e7d181120cfe687a519e2721f5ae92532d323533780241b0c: Status 404 returned error can't find the container with id 0347707d85d9423e7d181120cfe687a519e2721f5ae92532d323533780241b0c Sep 10 15:43:08.569062 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:08.569001 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-k8jsb" podStartSLOduration=3.084242383 podStartE2EDuration="4.568984614s" totalImagesPullingTime="1.414027103s" totalInitContainerRuntime="70.715128ms" isStatefulPod=true podCreationTimestamp="2026-09-10 15:43:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:43:08.567965556 +0000 UTC m=+65.921484712" watchObservedRunningTime="2026-09-10 15:43:08.568984614 +0000 UTC m=+65.922503775" Sep 10 15:43:08.628883 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:08.628821 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-bw9q7" podStartSLOduration=3.160578309 podStartE2EDuration="4.628798145s" totalImagesPullingTime="1.468219836s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:43:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:43:08.62831038 +0000 UTC m=+65.981829560" watchObservedRunningTime="2026-09-10 15:43:08.628798145 +0000 UTC m=+65.982317307" Sep 10 15:43:09.066966 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:09.066709 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6c668cb757-6pn84"] Sep 10 15:43:09.070102 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:09.070079 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6c668cb757-6pn84" Sep 10 15:43:09.072684 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:09.072659 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 10 15:43:09.072842 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:09.072659 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 10 15:43:09.072842 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:09.072662 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-3jrvdl6guu87v\"" Sep 10 15:43:09.073060 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:09.073037 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 10 15:43:09.073393 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:09.073361 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 10 15:43:09.073524 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:09.073448 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-dtgxm\"" Sep 10 15:43:09.081133 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:09.081096 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6c668cb757-6pn84"] Sep 10 15:43:09.174122 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:09.174088 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/494d55d5-ac80-4d7c-b9d1-a9541d59924f-audit-log\") pod \"metrics-server-6c668cb757-6pn84\" (UID: \"494d55d5-ac80-4d7c-b9d1-a9541d59924f\") " pod="openshift-monitoring/metrics-server-6c668cb757-6pn84" Sep 10 15:43:09.174122 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:09.174129 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/494d55d5-ac80-4d7c-b9d1-a9541d59924f-secret-metrics-server-tls\") pod \"metrics-server-6c668cb757-6pn84\" (UID: \"494d55d5-ac80-4d7c-b9d1-a9541d59924f\") " pod="openshift-monitoring/metrics-server-6c668cb757-6pn84" Sep 10 15:43:09.174349 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:09.174149 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/494d55d5-ac80-4d7c-b9d1-a9541d59924f-metrics-server-audit-profiles\") pod \"metrics-server-6c668cb757-6pn84\" (UID: \"494d55d5-ac80-4d7c-b9d1-a9541d59924f\") " pod="openshift-monitoring/metrics-server-6c668cb757-6pn84" Sep 10 15:43:09.174349 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:09.174178 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/494d55d5-ac80-4d7c-b9d1-a9541d59924f-secret-metrics-server-client-certs\") pod \"metrics-server-6c668cb757-6pn84\" (UID: \"494d55d5-ac80-4d7c-b9d1-a9541d59924f\") " pod="openshift-monitoring/metrics-server-6c668cb757-6pn84" Sep 10 15:43:09.174349 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:09.174196 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xfj8\" (UniqueName: \"kubernetes.io/projected/494d55d5-ac80-4d7c-b9d1-a9541d59924f-kube-api-access-2xfj8\") pod \"metrics-server-6c668cb757-6pn84\" (UID: \"494d55d5-ac80-4d7c-b9d1-a9541d59924f\") " pod="openshift-monitoring/metrics-server-6c668cb757-6pn84" Sep 10 15:43:09.174349 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:09.174219 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/494d55d5-ac80-4d7c-b9d1-a9541d59924f-client-ca-bundle\") pod \"metrics-server-6c668cb757-6pn84\" (UID: \"494d55d5-ac80-4d7c-b9d1-a9541d59924f\") " pod="openshift-monitoring/metrics-server-6c668cb757-6pn84" Sep 10 15:43:09.174349 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:09.174267 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/494d55d5-ac80-4d7c-b9d1-a9541d59924f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6c668cb757-6pn84\" (UID: \"494d55d5-ac80-4d7c-b9d1-a9541d59924f\") " pod="openshift-monitoring/metrics-server-6c668cb757-6pn84" Sep 10 15:43:09.274976 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:09.274937 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/494d55d5-ac80-4d7c-b9d1-a9541d59924f-secret-metrics-server-tls\") pod \"metrics-server-6c668cb757-6pn84\" (UID: \"494d55d5-ac80-4d7c-b9d1-a9541d59924f\") " pod="openshift-monitoring/metrics-server-6c668cb757-6pn84" Sep 10 15:43:09.275164 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:09.274990 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/494d55d5-ac80-4d7c-b9d1-a9541d59924f-metrics-server-audit-profiles\") pod \"metrics-server-6c668cb757-6pn84\" (UID: \"494d55d5-ac80-4d7c-b9d1-a9541d59924f\") " pod="openshift-monitoring/metrics-server-6c668cb757-6pn84" Sep 10 15:43:09.275164 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:09.275064 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/494d55d5-ac80-4d7c-b9d1-a9541d59924f-secret-metrics-server-client-certs\") pod \"metrics-server-6c668cb757-6pn84\" (UID: \"494d55d5-ac80-4d7c-b9d1-a9541d59924f\") " pod="openshift-monitoring/metrics-server-6c668cb757-6pn84" Sep 10 15:43:09.275164 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:09.275100 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xfj8\" (UniqueName: \"kubernetes.io/projected/494d55d5-ac80-4d7c-b9d1-a9541d59924f-kube-api-access-2xfj8\") pod \"metrics-server-6c668cb757-6pn84\" (UID: \"494d55d5-ac80-4d7c-b9d1-a9541d59924f\") " pod="openshift-monitoring/metrics-server-6c668cb757-6pn84" Sep 10 15:43:09.275164 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:09.275137 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/494d55d5-ac80-4d7c-b9d1-a9541d59924f-client-ca-bundle\") pod \"metrics-server-6c668cb757-6pn84\" (UID: \"494d55d5-ac80-4d7c-b9d1-a9541d59924f\") " pod="openshift-monitoring/metrics-server-6c668cb757-6pn84" Sep 10 15:43:09.275387 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:09.275178 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/494d55d5-ac80-4d7c-b9d1-a9541d59924f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6c668cb757-6pn84\" (UID: \"494d55d5-ac80-4d7c-b9d1-a9541d59924f\") " pod="openshift-monitoring/metrics-server-6c668cb757-6pn84" Sep 10 15:43:09.275387 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:09.275210 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/494d55d5-ac80-4d7c-b9d1-a9541d59924f-audit-log\") pod \"metrics-server-6c668cb757-6pn84\" (UID: \"494d55d5-ac80-4d7c-b9d1-a9541d59924f\") " pod="openshift-monitoring/metrics-server-6c668cb757-6pn84" Sep 10 15:43:09.275601 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:09.275567 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/494d55d5-ac80-4d7c-b9d1-a9541d59924f-audit-log\") pod \"metrics-server-6c668cb757-6pn84\" (UID: \"494d55d5-ac80-4d7c-b9d1-a9541d59924f\") " pod="openshift-monitoring/metrics-server-6c668cb757-6pn84" Sep 10 15:43:09.277058 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:09.277011 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/494d55d5-ac80-4d7c-b9d1-a9541d59924f-metrics-server-audit-profiles\") pod \"metrics-server-6c668cb757-6pn84\" (UID: \"494d55d5-ac80-4d7c-b9d1-a9541d59924f\") " pod="openshift-monitoring/metrics-server-6c668cb757-6pn84" Sep 10 15:43:09.277471 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:09.277434 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/494d55d5-ac80-4d7c-b9d1-a9541d59924f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6c668cb757-6pn84\" (UID: \"494d55d5-ac80-4d7c-b9d1-a9541d59924f\") " pod="openshift-monitoring/metrics-server-6c668cb757-6pn84" Sep 10 15:43:09.283040 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:09.282984 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/494d55d5-ac80-4d7c-b9d1-a9541d59924f-secret-metrics-server-tls\") pod \"metrics-server-6c668cb757-6pn84\" (UID: \"494d55d5-ac80-4d7c-b9d1-a9541d59924f\") " pod="openshift-monitoring/metrics-server-6c668cb757-6pn84" Sep 10 15:43:09.286241 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:09.286195 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xfj8\" (UniqueName: \"kubernetes.io/projected/494d55d5-ac80-4d7c-b9d1-a9541d59924f-kube-api-access-2xfj8\") pod \"metrics-server-6c668cb757-6pn84\" (UID: \"494d55d5-ac80-4d7c-b9d1-a9541d59924f\") " pod="openshift-monitoring/metrics-server-6c668cb757-6pn84" Sep 10 15:43:09.287868 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:09.287832 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/494d55d5-ac80-4d7c-b9d1-a9541d59924f-client-ca-bundle\") pod \"metrics-server-6c668cb757-6pn84\" (UID: \"494d55d5-ac80-4d7c-b9d1-a9541d59924f\") " pod="openshift-monitoring/metrics-server-6c668cb757-6pn84" Sep 10 15:43:09.290141 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:09.290116 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/494d55d5-ac80-4d7c-b9d1-a9541d59924f-secret-metrics-server-client-certs\") pod \"metrics-server-6c668cb757-6pn84\" (UID: \"494d55d5-ac80-4d7c-b9d1-a9541d59924f\") " pod="openshift-monitoring/metrics-server-6c668cb757-6pn84" Sep 10 15:43:09.382797 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:09.382695 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6c668cb757-6pn84" Sep 10 15:43:09.427079 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:09.426945 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:09.484804 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:09.484311 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:09.562061 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:09.562024 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9d4jc" event={"ID":"bbfbb1ad-2b61-4739-926f-01774a102c98","Type":"ContainerStarted","Data":"0347707d85d9423e7d181120cfe687a519e2721f5ae92532d323533780241b0c"} Sep 10 15:43:09.676779 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:09.676500 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6c668cb757-6pn84"] Sep 10 15:43:09.778460 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:43:09.778412 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod494d55d5_ac80_4d7c_b9d1_a9541d59924f.slice/crio-0ae044bb21355efd8edf497b01d8815e6bcc831cc78d8e77b0067ed82a34d223 WatchSource:0}: Error finding container 0ae044bb21355efd8edf497b01d8815e6bcc831cc78d8e77b0067ed82a34d223: Status 404 returned error can't find the container with id 0ae044bb21355efd8edf497b01d8815e6bcc831cc78d8e77b0067ed82a34d223 Sep 10 15:43:10.214837 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:10.214748 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:10.284621 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:10.284414 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:10.341363 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:10.341313 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:10.504397 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:10.503607 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-nt5ps" Sep 10 15:43:10.573220 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:10.573174 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9d4jc" event={"ID":"bbfbb1ad-2b61-4739-926f-01774a102c98","Type":"ContainerStarted","Data":"6f6b1a214a241b935194e20eb5ef6e482a6f226c12b98155afcfe9486e50c8ce"} Sep 10 15:43:10.573378 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:10.573244 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9d4jc" event={"ID":"bbfbb1ad-2b61-4739-926f-01774a102c98","Type":"ContainerStarted","Data":"ad27955df8799789323fd9c353669ab65ea59ff68e4c83d9efcd84d3a42eb4d8"} Sep 10 15:43:10.576255 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:10.576228 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6c668cb757-6pn84" event={"ID":"494d55d5-ac80-4d7c-b9d1-a9541d59924f","Type":"ContainerStarted","Data":"0ae044bb21355efd8edf497b01d8815e6bcc831cc78d8e77b0067ed82a34d223"} Sep 10 15:43:10.591540 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:10.591486 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-9d4jc" podStartSLOduration=65.327458708 podStartE2EDuration="1m6.591466132s" totalImagesPullingTime="1.264007424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:42:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:43:10.590892428 +0000 UTC m=+67.944411608" watchObservedRunningTime="2026-09-10 15:43:10.591466132 +0000 UTC m=+67.944985297" Sep 10 15:43:10.694891 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:10.694848 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:10.794558 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:10.794514 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:11.015191 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.015136 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 15:43:11.027237 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.027110 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:11.028828 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.028808 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:43:11.033179 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.033160 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 10 15:43:11.033396 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.033383 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 10 15:43:11.033573 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.033558 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 10 15:43:11.033651 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.033633 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 10 15:43:11.033937 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.033921 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 10 15:43:11.034025 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.034002 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 10 15:43:11.034186 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.034170 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 10 15:43:11.034272 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.034259 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 10 15:43:11.034359 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.034344 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-34cuaeejtnnt6\"" Sep 10 15:43:11.034412 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.034396 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 10 15:43:11.034589 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.034575 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 10 15:43:11.043565 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.043510 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-qhzpz\"" Sep 10 15:43:11.045061 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.045009 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 10 15:43:11.045177 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.045010 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 10 15:43:11.045844 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.045824 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 10 15:43:11.054930 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.054907 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 15:43:11.057687 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.057669 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 10 15:43:11.100612 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.100587 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b8e44c28-d988-4f54-8e09-363cf3cf55a2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:43:11.100784 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.100618 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b8e44c28-d988-4f54-8e09-363cf3cf55a2-web-config\") pod \"prometheus-k8s-0\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:43:11.100784 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.100635 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8e44c28-d988-4f54-8e09-363cf3cf55a2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:43:11.100784 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.100662 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b8e44c28-d988-4f54-8e09-363cf3cf55a2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:43:11.100784 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.100679 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b8e44c28-d988-4f54-8e09-363cf3cf55a2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:43:11.100784 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.100696 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b8e44c28-d988-4f54-8e09-363cf3cf55a2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:43:11.100784 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.100716 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b8e44c28-d988-4f54-8e09-363cf3cf55a2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:43:11.100784 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.100754 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b8e44c28-d988-4f54-8e09-363cf3cf55a2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:43:11.100784 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.100773 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b8e44c28-d988-4f54-8e09-363cf3cf55a2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:43:11.100784 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.100787 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8e44c28-d988-4f54-8e09-363cf3cf55a2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:43:11.101245 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.100807 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b8e44c28-d988-4f54-8e09-363cf3cf55a2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:43:11.101245 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.100837 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8e44c28-d988-4f54-8e09-363cf3cf55a2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:43:11.101245 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.100854 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b8e44c28-d988-4f54-8e09-363cf3cf55a2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:43:11.101245 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.100870 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b8e44c28-d988-4f54-8e09-363cf3cf55a2-config\") pod \"prometheus-k8s-0\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:43:11.101245 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.100903 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8e44c28-d988-4f54-8e09-363cf3cf55a2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:43:11.101245 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.100925 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b8e44c28-d988-4f54-8e09-363cf3cf55a2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:43:11.101245 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.100943 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpmgg\" (UniqueName: \"kubernetes.io/projected/b8e44c28-d988-4f54-8e09-363cf3cf55a2-kube-api-access-rpmgg\") pod \"prometheus-k8s-0\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:43:11.101245 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.100960 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b8e44c28-d988-4f54-8e09-363cf3cf55a2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:43:11.101245 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.100977 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b8e44c28-d988-4f54-8e09-363cf3cf55a2-config-out\") pod \"prometheus-k8s-0\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:43:11.101245 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.100994 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b8e44c28-d988-4f54-8e09-363cf3cf55a2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:43:11.201749 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.201707 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8e44c28-d988-4f54-8e09-363cf3cf55a2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:43:11.201908 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.201764 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b8e44c28-d988-4f54-8e09-363cf3cf55a2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:43:11.201908 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.201786 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpmgg\" (UniqueName: \"kubernetes.io/projected/b8e44c28-d988-4f54-8e09-363cf3cf55a2-kube-api-access-rpmgg\") pod \"prometheus-k8s-0\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:43:11.201908 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.201810 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b8e44c28-d988-4f54-8e09-363cf3cf55a2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:43:11.201908 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.201832 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b8e44c28-d988-4f54-8e09-363cf3cf55a2-config-out\") pod \"prometheus-k8s-0\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:43:11.201908 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.201854 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b8e44c28-d988-4f54-8e09-363cf3cf55a2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:43:11.201908 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.201888 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b8e44c28-d988-4f54-8e09-363cf3cf55a2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:43:11.203393 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.202305 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b8e44c28-d988-4f54-8e09-363cf3cf55a2-web-config\") pod \"prometheus-k8s-0\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:43:11.203393 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.202356 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8e44c28-d988-4f54-8e09-363cf3cf55a2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:43:11.203393 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.202416 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b8e44c28-d988-4f54-8e09-363cf3cf55a2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:43:11.203393 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.202452 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b8e44c28-d988-4f54-8e09-363cf3cf55a2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:43:11.203393 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.202500 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b8e44c28-d988-4f54-8e09-363cf3cf55a2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:43:11.203393 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.202536 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b8e44c28-d988-4f54-8e09-363cf3cf55a2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:43:11.203393 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.202565 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b8e44c28-d988-4f54-8e09-363cf3cf55a2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:43:11.203393 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.202600 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b8e44c28-d988-4f54-8e09-363cf3cf55a2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:43:11.203393 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.202626 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8e44c28-d988-4f54-8e09-363cf3cf55a2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:43:11.203393 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.202782 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b8e44c28-d988-4f54-8e09-363cf3cf55a2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:43:11.203393 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.202892 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8e44c28-d988-4f54-8e09-363cf3cf55a2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:43:11.203393 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.202626 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b8e44c28-d988-4f54-8e09-363cf3cf55a2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:43:11.203393 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.202925 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b8e44c28-d988-4f54-8e09-363cf3cf55a2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:43:11.203393 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.202959 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b8e44c28-d988-4f54-8e09-363cf3cf55a2-config\") pod \"prometheus-k8s-0\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:43:11.203393 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.202844 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8e44c28-d988-4f54-8e09-363cf3cf55a2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:43:11.204190 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.204161 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8e44c28-d988-4f54-8e09-363cf3cf55a2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:43:11.204252 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.202357 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b8e44c28-d988-4f54-8e09-363cf3cf55a2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:43:11.204934 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.204908 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b8e44c28-d988-4f54-8e09-363cf3cf55a2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:43:11.205027 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.204946 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8e44c28-d988-4f54-8e09-363cf3cf55a2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:43:11.208990 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.205610 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b8e44c28-d988-4f54-8e09-363cf3cf55a2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:43:11.208990 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.206189 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b8e44c28-d988-4f54-8e09-363cf3cf55a2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:43:11.208990 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.207225 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b8e44c28-d988-4f54-8e09-363cf3cf55a2-config-out\") pod \"prometheus-k8s-0\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:43:11.208990 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.207590 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b8e44c28-d988-4f54-8e09-363cf3cf55a2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:43:11.208990 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.207719 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b8e44c28-d988-4f54-8e09-363cf3cf55a2-config\") pod \"prometheus-k8s-0\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:43:11.208990 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.206344 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8e44c28-d988-4f54-8e09-363cf3cf55a2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:43:11.208990 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.208417 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b8e44c28-d988-4f54-8e09-363cf3cf55a2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:43:11.209371 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.209345 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b8e44c28-d988-4f54-8e09-363cf3cf55a2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:43:11.209686 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.209650 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b8e44c28-d988-4f54-8e09-363cf3cf55a2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:43:11.212216 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.212193 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b8e44c28-d988-4f54-8e09-363cf3cf55a2-web-config\") pod \"prometheus-k8s-0\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:43:11.214094 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.214062 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b8e44c28-d988-4f54-8e09-363cf3cf55a2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:43:11.214222 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.214203 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpmgg\" (UniqueName: \"kubernetes.io/projected/b8e44c28-d988-4f54-8e09-363cf3cf55a2-kube-api-access-rpmgg\") pod \"prometheus-k8s-0\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:43:11.215071 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.215051 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b8e44c28-d988-4f54-8e09-363cf3cf55a2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:43:11.215902 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.215881 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b8e44c28-d988-4f54-8e09-363cf3cf55a2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:43:11.343353 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:11.343299 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:43:14.003546 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:14.003512 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:14.058897 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:14.058874 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:14.363187 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:14.363160 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 15:43:14.367209 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:43:14.367183 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8e44c28_d988_4f54_8e09_363cf3cf55a2.slice/crio-d11e9c43b1d5209fc04670646a978c72c1db04a457d3cb6d98dc89a59ac09a0d WatchSource:0}: Error finding container d11e9c43b1d5209fc04670646a978c72c1db04a457d3cb6d98dc89a59ac09a0d: Status 404 returned error can't find the container with id d11e9c43b1d5209fc04670646a978c72c1db04a457d3cb6d98dc89a59ac09a0d Sep 10 15:43:14.452634 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:14.452603 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:14.553124 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:14.553089 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:14.593843 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:14.593716 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6c668cb757-6pn84" event={"ID":"494d55d5-ac80-4d7c-b9d1-a9541d59924f","Type":"ContainerStarted","Data":"43912e364341633d20c367073ee21a6c87509fd437eaf9775febb3760613d5be"} Sep 10 15:43:14.596159 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:14.596135 2576 generic.go:356] "Generic (PLEG): container finished" podID="b8e44c28-d988-4f54-8e09-363cf3cf55a2" containerID="11df3c11441b3cecabfedb1c89b5d6d61ab7351dbab4bab6b5e898a13abe0de4" exitCode=0 Sep 10 15:43:14.596275 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:14.596213 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8e44c28-d988-4f54-8e09-363cf3cf55a2","Type":"ContainerDied","Data":"11df3c11441b3cecabfedb1c89b5d6d61ab7351dbab4bab6b5e898a13abe0de4"} Sep 10 15:43:14.596342 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:14.596293 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8e44c28-d988-4f54-8e09-363cf3cf55a2","Type":"ContainerStarted","Data":"d11e9c43b1d5209fc04670646a978c72c1db04a457d3cb6d98dc89a59ac09a0d"} Sep 10 15:43:14.599925 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:14.599902 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb","Type":"ContainerStarted","Data":"7b268713962ccdd5b9a07d1cf32353b62779dc21365be75693e0a83ee9728a8c"} Sep 10 15:43:14.600023 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:14.599957 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb","Type":"ContainerStarted","Data":"0b77941fb12f20d963572ad3ae27e72beaa73075a9879fb3d8c660164b41149d"} Sep 10 15:43:14.600023 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:14.599995 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb","Type":"ContainerStarted","Data":"b6eeedab9e833977e99d59492dc5b9ae2868ef098b9909a92b24a09147552982"} Sep 10 15:43:14.600023 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:14.600020 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb","Type":"ContainerStarted","Data":"580c96036a76d764882779627b5d80e30a2fa6ef6ce760e88fb119b7d0c32a27"} Sep 10 15:43:14.600136 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:14.600047 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb","Type":"ContainerStarted","Data":"b15acc6ccc1036a8a2b74ba3d138e14aee1f71934e316e09413110d507c99f37"} Sep 10 15:43:14.601517 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:14.601498 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hzjqc" event={"ID":"1b1f777f-8ed4-408b-9721-80c9ae47af6e","Type":"ContainerStarted","Data":"0698050a50553ebe82d6897582fdc9ff8f0439f411db307b5e1ae6b2d7833a94"} Sep 10 15:43:14.603169 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:14.603147 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ptjhb" event={"ID":"c27ce7b1-1b71-4f92-8897-fd4748ebd262","Type":"ContainerStarted","Data":"e8c2186fb6b6b5d9a35a84448500d3510d9dfe5f4dad737d2a63cc38248ad40a"} Sep 10 15:43:14.603345 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:14.603328 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ptjhb" Sep 10 15:43:14.609964 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:14.609921 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6c668cb757-6pn84" podStartSLOduration=1.30791059 podStartE2EDuration="5.609905548s" totalImagesPullingTime="4.301994958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:43:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:43:14.609671756 +0000 UTC m=+71.963190935" watchObservedRunningTime="2026-09-10 15:43:14.609905548 +0000 UTC m=+71.963424709" Sep 10 15:43:14.650367 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:14.650322 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-hzjqc" podStartSLOduration=64.936018648 podStartE2EDuration="1m10.650307688s" totalImagesPullingTime="5.71428904s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:42:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:43:14.649221088 +0000 UTC m=+72.002740269" watchObservedRunningTime="2026-09-10 15:43:14.650307688 +0000 UTC m=+72.003826847" Sep 10 15:43:14.663295 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:14.663254 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-ptjhb" podStartSLOduration=64.966433646 podStartE2EDuration="1m10.663239823s" totalImagesPullingTime="5.696806177s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:42:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:43:14.662386643 +0000 UTC m=+72.015905816" watchObservedRunningTime="2026-09-10 15:43:14.663239823 +0000 UTC m=+72.016758985" Sep 10 15:43:15.612822 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:15.612780 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb","Type":"ContainerStarted","Data":"116305c9714a82aabe7de2930bb2f3eae8a29f2e3ab9a7a373bcce6b3b4221ef"} Sep 10 15:43:15.646885 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:15.646296 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.177093399 podStartE2EDuration="10.646277997s" totalImagesPullingTime="7.454198725s" totalInitContainerRuntime="14.985873ms" isStatefulPod=false podCreationTimestamp="2026-09-10 15:43:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:43:15.645459675 +0000 UTC m=+72.998978839" watchObservedRunningTime="2026-09-10 15:43:15.646277997 +0000 UTC m=+72.999797160" Sep 10 15:43:17.182561 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:17.182520 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:17.222009 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:17.221979 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-648dccb4c6-2nc2p"] Sep 10 15:43:17.226379 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:17.226354 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-648dccb4c6-2nc2p" Sep 10 15:43:17.239481 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:17.239459 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-648dccb4c6-2nc2p"] Sep 10 15:43:17.254707 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:17.254685 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3630e074-3c8f-49f1-8c97-0dab8c8d867b-console-config\") pod \"console-648dccb4c6-2nc2p\" (UID: \"3630e074-3c8f-49f1-8c97-0dab8c8d867b\") " pod="openshift-console/console-648dccb4c6-2nc2p" Sep 10 15:43:17.254851 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:17.254721 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxxkf\" (UniqueName: \"kubernetes.io/projected/3630e074-3c8f-49f1-8c97-0dab8c8d867b-kube-api-access-qxxkf\") pod \"console-648dccb4c6-2nc2p\" (UID: \"3630e074-3c8f-49f1-8c97-0dab8c8d867b\") " pod="openshift-console/console-648dccb4c6-2nc2p" Sep 10 15:43:17.254918 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:17.254844 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3630e074-3c8f-49f1-8c97-0dab8c8d867b-oauth-serving-cert\") pod \"console-648dccb4c6-2nc2p\" (UID: \"3630e074-3c8f-49f1-8c97-0dab8c8d867b\") " pod="openshift-console/console-648dccb4c6-2nc2p" Sep 10 15:43:17.254976 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:17.254952 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3630e074-3c8f-49f1-8c97-0dab8c8d867b-console-serving-cert\") pod \"console-648dccb4c6-2nc2p\" (UID: \"3630e074-3c8f-49f1-8c97-0dab8c8d867b\") " pod="openshift-console/console-648dccb4c6-2nc2p" Sep 10 15:43:17.255032 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:17.254998 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3630e074-3c8f-49f1-8c97-0dab8c8d867b-service-ca\") pod \"console-648dccb4c6-2nc2p\" (UID: \"3630e074-3c8f-49f1-8c97-0dab8c8d867b\") " pod="openshift-console/console-648dccb4c6-2nc2p" Sep 10 15:43:17.255083 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:17.255027 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3630e074-3c8f-49f1-8c97-0dab8c8d867b-console-oauth-config\") pod \"console-648dccb4c6-2nc2p\" (UID: \"3630e074-3c8f-49f1-8c97-0dab8c8d867b\") " pod="openshift-console/console-648dccb4c6-2nc2p" Sep 10 15:43:17.255083 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:17.255050 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3630e074-3c8f-49f1-8c97-0dab8c8d867b-trusted-ca-bundle\") pod \"console-648dccb4c6-2nc2p\" (UID: \"3630e074-3c8f-49f1-8c97-0dab8c8d867b\") " pod="openshift-console/console-648dccb4c6-2nc2p" Sep 10 15:43:17.320232 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:17.320198 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:17.356221 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:17.356190 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3630e074-3c8f-49f1-8c97-0dab8c8d867b-console-serving-cert\") pod \"console-648dccb4c6-2nc2p\" (UID: \"3630e074-3c8f-49f1-8c97-0dab8c8d867b\") " pod="openshift-console/console-648dccb4c6-2nc2p" Sep 10 15:43:17.356356 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:17.356231 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3630e074-3c8f-49f1-8c97-0dab8c8d867b-service-ca\") pod \"console-648dccb4c6-2nc2p\" (UID: \"3630e074-3c8f-49f1-8c97-0dab8c8d867b\") " pod="openshift-console/console-648dccb4c6-2nc2p" Sep 10 15:43:17.356356 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:17.356254 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3630e074-3c8f-49f1-8c97-0dab8c8d867b-console-oauth-config\") pod \"console-648dccb4c6-2nc2p\" (UID: \"3630e074-3c8f-49f1-8c97-0dab8c8d867b\") " pod="openshift-console/console-648dccb4c6-2nc2p" Sep 10 15:43:17.356356 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:17.356275 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3630e074-3c8f-49f1-8c97-0dab8c8d867b-trusted-ca-bundle\") pod \"console-648dccb4c6-2nc2p\" (UID: \"3630e074-3c8f-49f1-8c97-0dab8c8d867b\") " pod="openshift-console/console-648dccb4c6-2nc2p" Sep 10 15:43:17.356356 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:17.356328 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3630e074-3c8f-49f1-8c97-0dab8c8d867b-console-config\") pod \"console-648dccb4c6-2nc2p\" (UID: \"3630e074-3c8f-49f1-8c97-0dab8c8d867b\") " pod="openshift-console/console-648dccb4c6-2nc2p" Sep 10 15:43:17.356356 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:17.356352 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxxkf\" (UniqueName: \"kubernetes.io/projected/3630e074-3c8f-49f1-8c97-0dab8c8d867b-kube-api-access-qxxkf\") pod \"console-648dccb4c6-2nc2p\" (UID: \"3630e074-3c8f-49f1-8c97-0dab8c8d867b\") " pod="openshift-console/console-648dccb4c6-2nc2p" Sep 10 15:43:17.356600 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:17.356409 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3630e074-3c8f-49f1-8c97-0dab8c8d867b-oauth-serving-cert\") pod \"console-648dccb4c6-2nc2p\" (UID: \"3630e074-3c8f-49f1-8c97-0dab8c8d867b\") " pod="openshift-console/console-648dccb4c6-2nc2p" Sep 10 15:43:17.357031 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:17.357008 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3630e074-3c8f-49f1-8c97-0dab8c8d867b-service-ca\") pod \"console-648dccb4c6-2nc2p\" (UID: \"3630e074-3c8f-49f1-8c97-0dab8c8d867b\") " pod="openshift-console/console-648dccb4c6-2nc2p" Sep 10 15:43:17.357031 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:17.357024 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3630e074-3c8f-49f1-8c97-0dab8c8d867b-console-config\") pod \"console-648dccb4c6-2nc2p\" (UID: \"3630e074-3c8f-49f1-8c97-0dab8c8d867b\") " pod="openshift-console/console-648dccb4c6-2nc2p" Sep 10 15:43:17.357211 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:17.357107 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3630e074-3c8f-49f1-8c97-0dab8c8d867b-oauth-serving-cert\") pod \"console-648dccb4c6-2nc2p\" (UID: \"3630e074-3c8f-49f1-8c97-0dab8c8d867b\") " pod="openshift-console/console-648dccb4c6-2nc2p" Sep 10 15:43:17.357541 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:17.357519 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3630e074-3c8f-49f1-8c97-0dab8c8d867b-trusted-ca-bundle\") pod \"console-648dccb4c6-2nc2p\" (UID: \"3630e074-3c8f-49f1-8c97-0dab8c8d867b\") " pod="openshift-console/console-648dccb4c6-2nc2p" Sep 10 15:43:17.359539 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:17.359278 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3630e074-3c8f-49f1-8c97-0dab8c8d867b-console-oauth-config\") pod \"console-648dccb4c6-2nc2p\" (UID: \"3630e074-3c8f-49f1-8c97-0dab8c8d867b\") " pod="openshift-console/console-648dccb4c6-2nc2p" Sep 10 15:43:17.359794 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:17.359768 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3630e074-3c8f-49f1-8c97-0dab8c8d867b-console-serving-cert\") pod \"console-648dccb4c6-2nc2p\" (UID: \"3630e074-3c8f-49f1-8c97-0dab8c8d867b\") " pod="openshift-console/console-648dccb4c6-2nc2p" Sep 10 15:43:17.366805 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:17.366780 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxxkf\" (UniqueName: \"kubernetes.io/projected/3630e074-3c8f-49f1-8c97-0dab8c8d867b-kube-api-access-qxxkf\") pod \"console-648dccb4c6-2nc2p\" (UID: \"3630e074-3c8f-49f1-8c97-0dab8c8d867b\") " pod="openshift-console/console-648dccb4c6-2nc2p" Sep 10 15:43:17.537119 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:17.537033 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-648dccb4c6-2nc2p" Sep 10 15:43:17.959354 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:17.959317 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-648dccb4c6-2nc2p"] Sep 10 15:43:17.976428 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:43:17.976395 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3630e074_3c8f_49f1_8c97_0dab8c8d867b.slice/crio-777f76761069c4b3dc6bba5f43b95a02c5c1c009b76f08512c2608d1655cbb99 WatchSource:0}: Error finding container 777f76761069c4b3dc6bba5f43b95a02c5c1c009b76f08512c2608d1655cbb99: Status 404 returned error can't find the container with id 777f76761069c4b3dc6bba5f43b95a02c5c1c009b76f08512c2608d1655cbb99 Sep 10 15:43:18.269363 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:18.269324 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:18.421672 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:18.421591 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:18.529119 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:18.529073 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:18.624593 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:18.623853 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:18.629743 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:18.629698 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8e44c28-d988-4f54-8e09-363cf3cf55a2","Type":"ContainerStarted","Data":"d1924b4ff00fda092ad255cdbabcce72bac71ace4a3460c4790268e73ee7876f"} Sep 10 15:43:18.629918 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:18.629783 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8e44c28-d988-4f54-8e09-363cf3cf55a2","Type":"ContainerStarted","Data":"418ce35d1d2fcf9b9456e40180323d30ba760be7d12e5b8c72d41ccaed4d646b"} Sep 10 15:43:18.631517 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:18.631493 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-648dccb4c6-2nc2p" event={"ID":"3630e074-3c8f-49f1-8c97-0dab8c8d867b","Type":"ContainerStarted","Data":"3102422383c1aae668fc56b14416b61d6a24fe4d85e7441ace75398e6655371b"} Sep 10 15:43:18.631605 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:18.631538 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-648dccb4c6-2nc2p" event={"ID":"3630e074-3c8f-49f1-8c97-0dab8c8d867b","Type":"ContainerStarted","Data":"777f76761069c4b3dc6bba5f43b95a02c5c1c009b76f08512c2608d1655cbb99"} Sep 10 15:43:18.651786 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:18.651721 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-648dccb4c6-2nc2p" podStartSLOduration=1.6517052909999999 podStartE2EDuration="1.651705291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:43:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:43:18.649837357 +0000 UTC m=+76.003356535" watchObservedRunningTime="2026-09-10 15:43:18.651705291 +0000 UTC m=+76.005224452" Sep 10 15:43:19.483329 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:19.483282 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:19.576052 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:19.576021 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:19.639414 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:19.639369 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8e44c28-d988-4f54-8e09-363cf3cf55a2","Type":"ContainerStarted","Data":"45c12b3a07236dd8d6ab3511e23ec70dc18940a6176c72924e85cea49f2a5d95"} Sep 10 15:43:20.647047 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:20.647013 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8e44c28-d988-4f54-8e09-363cf3cf55a2","Type":"ContainerStarted","Data":"5ce4293653053039cc902f4a45944b9c3c71c680f7fc5d47d80d8f57e8a11ffb"} Sep 10 15:43:20.647453 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:20.647089 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8e44c28-d988-4f54-8e09-363cf3cf55a2","Type":"ContainerStarted","Data":"c9fbed8a5161c1472f48249a5be47597df0e77e8d6e2085faa723eebdcececf1"} Sep 10 15:43:20.647453 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:20.647128 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8e44c28-d988-4f54-8e09-363cf3cf55a2","Type":"ContainerStarted","Data":"9a1afa6f48a3cba3f2adfda5f597365bb7c650932bb809e96e612ae9929bbdb4"} Sep 10 15:43:20.684810 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:20.684750 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=5.83856873 podStartE2EDuration="10.684712763s" totalImagesPullingTime="4.82451455s" totalInitContainerRuntime="21.629483ms" isStatefulPod=false podCreationTimestamp="2026-09-10 15:43:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:43:20.682431695 +0000 UTC m=+78.035950882" watchObservedRunningTime="2026-09-10 15:43:20.684712763 +0000 UTC m=+78.038231925" Sep 10 15:43:21.344191 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:21.344150 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:43:23.090652 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:23.090614 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:23.190984 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:23.190956 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:24.471367 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:24.471227 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:24.572329 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:24.572292 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:27.538169 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:27.538134 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-648dccb4c6-2nc2p" Sep 10 15:43:27.538785 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:27.538215 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-648dccb4c6-2nc2p" Sep 10 15:43:27.543378 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:27.543351 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-648dccb4c6-2nc2p" Sep 10 15:43:27.682412 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:27.682385 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-648dccb4c6-2nc2p" Sep 10 15:43:27.728461 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:27.728425 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d67c6549f-t9kz4"] Sep 10 15:43:27.840461 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:27.840419 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:27.942582 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:27.942541 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:28.007835 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:28.007804 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:29.383641 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:29.383603 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6c668cb757-6pn84" Sep 10 15:43:29.384042 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:29.383669 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6c668cb757-6pn84" Sep 10 15:43:32.189993 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:32.189945 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:32.307316 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:32.307287 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:35.083377 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:35.083340 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:35.148770 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:35.148719 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:36.492185 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:36.492147 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:41.795473 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:41.795430 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:41.854316 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:41.854268 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:44.263816 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:44.263778 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:44.364930 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:44.364891 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:44.421133 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:44.421080 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:45.614941 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:45.614915 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-ptjhb" Sep 10 15:43:49.388942 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:49.388913 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6c668cb757-6pn84" Sep 10 15:43:49.392843 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:49.392820 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6c668cb757-6pn84" Sep 10 15:43:52.749583 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:52.749516 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5d67c6549f-t9kz4" podUID="af48efd8-d70a-4bc3-9f4f-250ce098c8e6" containerName="console" containerID="cri-o://52cb12023a69aaa97215810660cfd699027acfbb9910bdeb7db1629238e38caa" gracePeriod=15 Sep 10 15:43:52.993881 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:52.993860 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d67c6549f-t9kz4" Sep 10 15:43:53.049415 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:53.049389 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tmt4v\" (UniqueName: \"kubernetes.io/projected/af48efd8-d70a-4bc3-9f4f-250ce098c8e6-kube-api-access-tmt4v\") pod \"af48efd8-d70a-4bc3-9f4f-250ce098c8e6\" (UID: \"af48efd8-d70a-4bc3-9f4f-250ce098c8e6\") " Sep 10 15:43:53.049532 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:53.049439 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af48efd8-d70a-4bc3-9f4f-250ce098c8e6-trusted-ca-bundle\") pod \"af48efd8-d70a-4bc3-9f4f-250ce098c8e6\" (UID: \"af48efd8-d70a-4bc3-9f4f-250ce098c8e6\") " Sep 10 15:43:53.049532 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:53.049457 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/af48efd8-d70a-4bc3-9f4f-250ce098c8e6-console-config\") pod \"af48efd8-d70a-4bc3-9f4f-250ce098c8e6\" (UID: \"af48efd8-d70a-4bc3-9f4f-250ce098c8e6\") " Sep 10 15:43:53.049532 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:53.049476 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/af48efd8-d70a-4bc3-9f4f-250ce098c8e6-console-oauth-config\") pod \"af48efd8-d70a-4bc3-9f4f-250ce098c8e6\" (UID: \"af48efd8-d70a-4bc3-9f4f-250ce098c8e6\") " Sep 10 15:43:53.049532 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:53.049500 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/af48efd8-d70a-4bc3-9f4f-250ce098c8e6-service-ca\") pod \"af48efd8-d70a-4bc3-9f4f-250ce098c8e6\" (UID: \"af48efd8-d70a-4bc3-9f4f-250ce098c8e6\") " Sep 10 15:43:53.049532 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:53.049525 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/af48efd8-d70a-4bc3-9f4f-250ce098c8e6-oauth-serving-cert\") pod \"af48efd8-d70a-4bc3-9f4f-250ce098c8e6\" (UID: \"af48efd8-d70a-4bc3-9f4f-250ce098c8e6\") " Sep 10 15:43:53.049814 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:53.049555 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/af48efd8-d70a-4bc3-9f4f-250ce098c8e6-console-serving-cert\") pod \"af48efd8-d70a-4bc3-9f4f-250ce098c8e6\" (UID: \"af48efd8-d70a-4bc3-9f4f-250ce098c8e6\") " Sep 10 15:43:53.049902 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:53.049876 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af48efd8-d70a-4bc3-9f4f-250ce098c8e6-console-config" (OuterVolumeSpecName: "console-config") pod "af48efd8-d70a-4bc3-9f4f-250ce098c8e6" (UID: "af48efd8-d70a-4bc3-9f4f-250ce098c8e6"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:43:53.049952 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:53.049910 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af48efd8-d70a-4bc3-9f4f-250ce098c8e6-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "af48efd8-d70a-4bc3-9f4f-250ce098c8e6" (UID: "af48efd8-d70a-4bc3-9f4f-250ce098c8e6"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:43:53.049952 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:53.049929 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af48efd8-d70a-4bc3-9f4f-250ce098c8e6-service-ca" (OuterVolumeSpecName: "service-ca") pod "af48efd8-d70a-4bc3-9f4f-250ce098c8e6" (UID: "af48efd8-d70a-4bc3-9f4f-250ce098c8e6"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:43:53.049952 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:53.049939 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af48efd8-d70a-4bc3-9f4f-250ce098c8e6-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "af48efd8-d70a-4bc3-9f4f-250ce098c8e6" (UID: "af48efd8-d70a-4bc3-9f4f-250ce098c8e6"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:43:53.051863 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:53.051837 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af48efd8-d70a-4bc3-9f4f-250ce098c8e6-kube-api-access-tmt4v" (OuterVolumeSpecName: "kube-api-access-tmt4v") pod "af48efd8-d70a-4bc3-9f4f-250ce098c8e6" (UID: "af48efd8-d70a-4bc3-9f4f-250ce098c8e6"). InnerVolumeSpecName "kube-api-access-tmt4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:43:53.051963 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:53.051928 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af48efd8-d70a-4bc3-9f4f-250ce098c8e6-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "af48efd8-d70a-4bc3-9f4f-250ce098c8e6" (UID: "af48efd8-d70a-4bc3-9f4f-250ce098c8e6"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:43:53.052030 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:53.052001 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af48efd8-d70a-4bc3-9f4f-250ce098c8e6-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "af48efd8-d70a-4bc3-9f4f-250ce098c8e6" (UID: "af48efd8-d70a-4bc3-9f4f-250ce098c8e6"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:43:53.150153 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:53.150121 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmt4v\" (UniqueName: \"kubernetes.io/projected/af48efd8-d70a-4bc3-9f4f-250ce098c8e6-kube-api-access-tmt4v\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:43:53.150153 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:53.150155 2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af48efd8-d70a-4bc3-9f4f-250ce098c8e6-trusted-ca-bundle\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:43:53.150153 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:53.150170 2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/af48efd8-d70a-4bc3-9f4f-250ce098c8e6-console-config\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:43:53.150405 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:53.150185 2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/af48efd8-d70a-4bc3-9f4f-250ce098c8e6-console-oauth-config\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:43:53.150405 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:53.150198 2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/af48efd8-d70a-4bc3-9f4f-250ce098c8e6-service-ca\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:43:53.150405 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:53.150214 2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/af48efd8-d70a-4bc3-9f4f-250ce098c8e6-oauth-serving-cert\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:43:53.150405 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:53.150226 2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/af48efd8-d70a-4bc3-9f4f-250ce098c8e6-console-serving-cert\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:43:53.776409 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:53.776373 2576 generic.go:356] "Generic (PLEG): container finished" podID="af48efd8-d70a-4bc3-9f4f-250ce098c8e6" containerID="52cb12023a69aaa97215810660cfd699027acfbb9910bdeb7db1629238e38caa" exitCode=0 Sep 10 15:43:53.776833 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:53.776432 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d67c6549f-t9kz4" Sep 10 15:43:53.776833 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:53.776458 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d67c6549f-t9kz4" event={"ID":"af48efd8-d70a-4bc3-9f4f-250ce098c8e6","Type":"ContainerDied","Data":"52cb12023a69aaa97215810660cfd699027acfbb9910bdeb7db1629238e38caa"} Sep 10 15:43:53.776833 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:53.776526 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d67c6549f-t9kz4" event={"ID":"af48efd8-d70a-4bc3-9f4f-250ce098c8e6","Type":"ContainerDied","Data":"cbf9357a8da3e2663a1b3247e70728e94128599eaaf4f4a1d46ea43f02b22b79"} Sep 10 15:43:53.776833 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:53.776547 2576 scope.go:117] "RemoveContainer" containerID="52cb12023a69aaa97215810660cfd699027acfbb9910bdeb7db1629238e38caa" Sep 10 15:43:53.785468 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:53.785447 2576 scope.go:117] "RemoveContainer" containerID="52cb12023a69aaa97215810660cfd699027acfbb9910bdeb7db1629238e38caa" Sep 10 15:43:53.785787 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:43:53.785762 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52cb12023a69aaa97215810660cfd699027acfbb9910bdeb7db1629238e38caa\": container with ID starting with 52cb12023a69aaa97215810660cfd699027acfbb9910bdeb7db1629238e38caa not found: ID does not exist" containerID="52cb12023a69aaa97215810660cfd699027acfbb9910bdeb7db1629238e38caa" Sep 10 15:43:53.785848 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:53.785800 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52cb12023a69aaa97215810660cfd699027acfbb9910bdeb7db1629238e38caa"} err="failed to get container status \"52cb12023a69aaa97215810660cfd699027acfbb9910bdeb7db1629238e38caa\": rpc error: code = NotFound desc = could not find container \"52cb12023a69aaa97215810660cfd699027acfbb9910bdeb7db1629238e38caa\": container with ID starting with 52cb12023a69aaa97215810660cfd699027acfbb9910bdeb7db1629238e38caa not found: ID does not exist" Sep 10 15:43:53.793464 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:53.793440 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d67c6549f-t9kz4"] Sep 10 15:43:53.797205 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:53.797178 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5d67c6549f-t9kz4"] Sep 10 15:43:55.167494 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:43:55.167463 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af48efd8-d70a-4bc3-9f4f-250ce098c8e6" path="/var/lib/kubelet/pods/af48efd8-d70a-4bc3-9f4f-250ce098c8e6/volumes" Sep 10 15:44:11.344363 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:11.344326 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:11.364075 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:11.364050 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:11.870281 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:11.870254 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:24.012526 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:24.012493 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:44:25.052149 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:25.052113 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:44:25.070988 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:25.070957 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 15:44:25.071302 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:25.071279 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d9dd9c02-cf57-49d8-a55c-0139a15c7eeb" containerName="alertmanager" containerID="cri-o://b15acc6ccc1036a8a2b74ba3d138e14aee1f71934e316e09413110d507c99f37" gracePeriod=120 Sep 10 15:44:25.071372 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:25.071344 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d9dd9c02-cf57-49d8-a55c-0139a15c7eeb" containerName="kube-rbac-proxy-metric" containerID="cri-o://7b268713962ccdd5b9a07d1cf32353b62779dc21365be75693e0a83ee9728a8c" gracePeriod=120 Sep 10 15:44:25.071416 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:25.071375 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d9dd9c02-cf57-49d8-a55c-0139a15c7eeb" containerName="kube-rbac-proxy" containerID="cri-o://0b77941fb12f20d963572ad3ae27e72beaa73075a9879fb3d8c660164b41149d" gracePeriod=120 Sep 10 15:44:25.071471 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:25.071404 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d9dd9c02-cf57-49d8-a55c-0139a15c7eeb" containerName="config-reloader" containerID="cri-o://580c96036a76d764882779627b5d80e30a2fa6ef6ce760e88fb119b7d0c32a27" gracePeriod=120 Sep 10 15:44:25.071471 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:25.071421 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d9dd9c02-cf57-49d8-a55c-0139a15c7eeb" containerName="prom-label-proxy" containerID="cri-o://116305c9714a82aabe7de2930bb2f3eae8a29f2e3ab9a7a373bcce6b3b4221ef" gracePeriod=120 Sep 10 15:44:25.071471 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:25.071351 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d9dd9c02-cf57-49d8-a55c-0139a15c7eeb" containerName="kube-rbac-proxy-web" containerID="cri-o://b6eeedab9e833977e99d59492dc5b9ae2868ef098b9909a92b24a09147552982" gracePeriod=120 Sep 10 15:44:25.919614 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:25.919580 2576 generic.go:356] "Generic (PLEG): container finished" podID="d9dd9c02-cf57-49d8-a55c-0139a15c7eeb" containerID="116305c9714a82aabe7de2930bb2f3eae8a29f2e3ab9a7a373bcce6b3b4221ef" exitCode=0 Sep 10 15:44:25.919614 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:25.919607 2576 generic.go:356] "Generic (PLEG): container finished" podID="d9dd9c02-cf57-49d8-a55c-0139a15c7eeb" containerID="7b268713962ccdd5b9a07d1cf32353b62779dc21365be75693e0a83ee9728a8c" exitCode=0 Sep 10 15:44:25.919614 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:25.919613 2576 generic.go:356] "Generic (PLEG): container finished" podID="d9dd9c02-cf57-49d8-a55c-0139a15c7eeb" containerID="0b77941fb12f20d963572ad3ae27e72beaa73075a9879fb3d8c660164b41149d" exitCode=0 Sep 10 15:44:25.919614 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:25.919619 2576 generic.go:356] "Generic (PLEG): container finished" podID="d9dd9c02-cf57-49d8-a55c-0139a15c7eeb" containerID="580c96036a76d764882779627b5d80e30a2fa6ef6ce760e88fb119b7d0c32a27" exitCode=0 Sep 10 15:44:25.919614 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:25.919624 2576 generic.go:356] "Generic (PLEG): container finished" podID="d9dd9c02-cf57-49d8-a55c-0139a15c7eeb" containerID="b15acc6ccc1036a8a2b74ba3d138e14aee1f71934e316e09413110d507c99f37" exitCode=0 Sep 10 15:44:25.919927 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:25.919654 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb","Type":"ContainerDied","Data":"116305c9714a82aabe7de2930bb2f3eae8a29f2e3ab9a7a373bcce6b3b4221ef"} Sep 10 15:44:25.919927 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:25.919759 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb","Type":"ContainerDied","Data":"7b268713962ccdd5b9a07d1cf32353b62779dc21365be75693e0a83ee9728a8c"} Sep 10 15:44:25.919927 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:25.919789 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb","Type":"ContainerDied","Data":"0b77941fb12f20d963572ad3ae27e72beaa73075a9879fb3d8c660164b41149d"} Sep 10 15:44:25.919927 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:25.919815 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb","Type":"ContainerDied","Data":"580c96036a76d764882779627b5d80e30a2fa6ef6ce760e88fb119b7d0c32a27"} Sep 10 15:44:25.919927 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:25.919848 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb","Type":"ContainerDied","Data":"b15acc6ccc1036a8a2b74ba3d138e14aee1f71934e316e09413110d507c99f37"} Sep 10 15:44:26.312235 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.312211 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:44:26.416704 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.416675 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-secret-alertmanager-kube-rbac-proxy-web\") pod \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\" (UID: \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\") " Sep 10 15:44:26.416871 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.416713 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-config-volume\") pod \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\" (UID: \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\") " Sep 10 15:44:26.416871 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.416774 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-tls-assets\") pod \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\" (UID: \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\") " Sep 10 15:44:26.416871 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.416799 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\" (UID: \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\") " Sep 10 15:44:26.416871 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.416827 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-web-config\") pod \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\" (UID: \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\") " Sep 10 15:44:26.416871 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.416862 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-metrics-client-ca\") pod \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\" (UID: \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\") " Sep 10 15:44:26.417184 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.416895 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-secret-alertmanager-kube-rbac-proxy\") pod \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\" (UID: \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\") " Sep 10 15:44:26.417184 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.416919 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rvp7v\" (UniqueName: \"kubernetes.io/projected/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-kube-api-access-rvp7v\") pod \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\" (UID: \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\") " Sep 10 15:44:26.417184 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.416964 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-secret-alertmanager-main-tls\") pod \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\" (UID: \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\") " Sep 10 15:44:26.417184 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.417009 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-alertmanager-main-db\") pod \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\" (UID: \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\") " Sep 10 15:44:26.417184 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.417035 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-config-out\") pod \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\" (UID: \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\") " Sep 10 15:44:26.417184 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.417074 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-alertmanager-trusted-ca-bundle\") pod \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\" (UID: \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\") " Sep 10 15:44:26.417184 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.417099 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-cluster-tls-config\") pod \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\" (UID: \"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb\") " Sep 10 15:44:26.417692 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.417281 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "d9dd9c02-cf57-49d8-a55c-0139a15c7eeb" (UID: "d9dd9c02-cf57-49d8-a55c-0139a15c7eeb"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:44:26.417824 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.417799 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "d9dd9c02-cf57-49d8-a55c-0139a15c7eeb" (UID: "d9dd9c02-cf57-49d8-a55c-0139a15c7eeb"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:44:26.418565 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.418520 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "d9dd9c02-cf57-49d8-a55c-0139a15c7eeb" (UID: "d9dd9c02-cf57-49d8-a55c-0139a15c7eeb"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:44:26.419927 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.419888 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "d9dd9c02-cf57-49d8-a55c-0139a15c7eeb" (UID: "d9dd9c02-cf57-49d8-a55c-0139a15c7eeb"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:44:26.420108 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.420068 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-config-volume" (OuterVolumeSpecName: "config-volume") pod "d9dd9c02-cf57-49d8-a55c-0139a15c7eeb" (UID: "d9dd9c02-cf57-49d8-a55c-0139a15c7eeb"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:44:26.420306 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.420277 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "d9dd9c02-cf57-49d8-a55c-0139a15c7eeb" (UID: "d9dd9c02-cf57-49d8-a55c-0139a15c7eeb"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:44:26.420677 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.420646 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "d9dd9c02-cf57-49d8-a55c-0139a15c7eeb" (UID: "d9dd9c02-cf57-49d8-a55c-0139a15c7eeb"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:44:26.420919 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.420900 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "d9dd9c02-cf57-49d8-a55c-0139a15c7eeb" (UID: "d9dd9c02-cf57-49d8-a55c-0139a15c7eeb"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:44:26.421108 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.421079 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-config-out" (OuterVolumeSpecName: "config-out") pod "d9dd9c02-cf57-49d8-a55c-0139a15c7eeb" (UID: "d9dd9c02-cf57-49d8-a55c-0139a15c7eeb"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:44:26.421457 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.421434 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "d9dd9c02-cf57-49d8-a55c-0139a15c7eeb" (UID: "d9dd9c02-cf57-49d8-a55c-0139a15c7eeb"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:44:26.421537 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.421497 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-kube-api-access-rvp7v" (OuterVolumeSpecName: "kube-api-access-rvp7v") pod "d9dd9c02-cf57-49d8-a55c-0139a15c7eeb" (UID: "d9dd9c02-cf57-49d8-a55c-0139a15c7eeb"). InnerVolumeSpecName "kube-api-access-rvp7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:44:26.425813 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.425787 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "d9dd9c02-cf57-49d8-a55c-0139a15c7eeb" (UID: "d9dd9c02-cf57-49d8-a55c-0139a15c7eeb"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:44:26.430199 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.430173 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-web-config" (OuterVolumeSpecName: "web-config") pod "d9dd9c02-cf57-49d8-a55c-0139a15c7eeb" (UID: "d9dd9c02-cf57-49d8-a55c-0139a15c7eeb"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:44:26.518486 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.518437 2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:44:26.518486 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.518456 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvp7v\" (UniqueName: \"kubernetes.io/projected/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-kube-api-access-rvp7v\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:44:26.518486 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.518466 2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-secret-alertmanager-main-tls\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:44:26.518486 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.518475 2576 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-alertmanager-main-db\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:44:26.518486 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.518483 2576 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-config-out\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:44:26.518685 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.518491 2576 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:44:26.518685 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.518499 2576 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-cluster-tls-config\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:44:26.518685 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.518507 2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:44:26.518685 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.518516 2576 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-config-volume\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:44:26.518685 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.518523 2576 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-tls-assets\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:44:26.518685 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.518531 2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:44:26.518685 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.518540 2576 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-web-config\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:44:26.518685 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.518555 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb-metrics-client-ca\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:44:26.926820 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.926780 2576 generic.go:356] "Generic (PLEG): container finished" podID="d9dd9c02-cf57-49d8-a55c-0139a15c7eeb" containerID="b6eeedab9e833977e99d59492dc5b9ae2868ef098b9909a92b24a09147552982" exitCode=0 Sep 10 15:44:26.926991 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.926862 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb","Type":"ContainerDied","Data":"b6eeedab9e833977e99d59492dc5b9ae2868ef098b9909a92b24a09147552982"} Sep 10 15:44:26.926991 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.926934 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d9dd9c02-cf57-49d8-a55c-0139a15c7eeb","Type":"ContainerDied","Data":"f34a91977ceef34c5aace658054d9bccffe1bd86bc0ba402165193a583f08523"} Sep 10 15:44:26.926991 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.926875 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:44:26.926991 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.926965 2576 scope.go:117] "RemoveContainer" containerID="116305c9714a82aabe7de2930bb2f3eae8a29f2e3ab9a7a373bcce6b3b4221ef" Sep 10 15:44:26.934655 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.934639 2576 scope.go:117] "RemoveContainer" containerID="7b268713962ccdd5b9a07d1cf32353b62779dc21365be75693e0a83ee9728a8c" Sep 10 15:44:26.941518 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.941505 2576 scope.go:117] "RemoveContainer" containerID="0b77941fb12f20d963572ad3ae27e72beaa73075a9879fb3d8c660164b41149d" Sep 10 15:44:26.948023 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.948002 2576 scope.go:117] "RemoveContainer" containerID="b6eeedab9e833977e99d59492dc5b9ae2868ef098b9909a92b24a09147552982" Sep 10 15:44:26.951043 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.951021 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 15:44:26.954991 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.954964 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 15:44:26.955890 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.955870 2576 scope.go:117] "RemoveContainer" containerID="580c96036a76d764882779627b5d80e30a2fa6ef6ce760e88fb119b7d0c32a27" Sep 10 15:44:26.962820 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.962798 2576 scope.go:117] "RemoveContainer" containerID="b15acc6ccc1036a8a2b74ba3d138e14aee1f71934e316e09413110d507c99f37" Sep 10 15:44:26.969488 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.969470 2576 scope.go:117] "RemoveContainer" containerID="2bebb7dcd3bb8af8304a2d5b718d5a84a065932fddb60ce095460bb246dce756" Sep 10 15:44:26.976255 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.976241 2576 scope.go:117] "RemoveContainer" containerID="116305c9714a82aabe7de2930bb2f3eae8a29f2e3ab9a7a373bcce6b3b4221ef" Sep 10 15:44:26.976502 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:44:26.976482 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"116305c9714a82aabe7de2930bb2f3eae8a29f2e3ab9a7a373bcce6b3b4221ef\": container with ID starting with 116305c9714a82aabe7de2930bb2f3eae8a29f2e3ab9a7a373bcce6b3b4221ef not found: ID does not exist" containerID="116305c9714a82aabe7de2930bb2f3eae8a29f2e3ab9a7a373bcce6b3b4221ef" Sep 10 15:44:26.976564 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.976509 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"116305c9714a82aabe7de2930bb2f3eae8a29f2e3ab9a7a373bcce6b3b4221ef"} err="failed to get container status \"116305c9714a82aabe7de2930bb2f3eae8a29f2e3ab9a7a373bcce6b3b4221ef\": rpc error: code = NotFound desc = could not find container \"116305c9714a82aabe7de2930bb2f3eae8a29f2e3ab9a7a373bcce6b3b4221ef\": container with ID starting with 116305c9714a82aabe7de2930bb2f3eae8a29f2e3ab9a7a373bcce6b3b4221ef not found: ID does not exist" Sep 10 15:44:26.976564 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.976527 2576 scope.go:117] "RemoveContainer" containerID="7b268713962ccdd5b9a07d1cf32353b62779dc21365be75693e0a83ee9728a8c" Sep 10 15:44:26.976835 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:44:26.976727 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b268713962ccdd5b9a07d1cf32353b62779dc21365be75693e0a83ee9728a8c\": container with ID starting with 7b268713962ccdd5b9a07d1cf32353b62779dc21365be75693e0a83ee9728a8c not found: ID does not exist" containerID="7b268713962ccdd5b9a07d1cf32353b62779dc21365be75693e0a83ee9728a8c" Sep 10 15:44:26.976878 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.976839 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b268713962ccdd5b9a07d1cf32353b62779dc21365be75693e0a83ee9728a8c"} err="failed to get container status \"7b268713962ccdd5b9a07d1cf32353b62779dc21365be75693e0a83ee9728a8c\": rpc error: code = NotFound desc = could not find container \"7b268713962ccdd5b9a07d1cf32353b62779dc21365be75693e0a83ee9728a8c\": container with ID starting with 7b268713962ccdd5b9a07d1cf32353b62779dc21365be75693e0a83ee9728a8c not found: ID does not exist" Sep 10 15:44:26.976878 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.976851 2576 scope.go:117] "RemoveContainer" containerID="0b77941fb12f20d963572ad3ae27e72beaa73075a9879fb3d8c660164b41149d" Sep 10 15:44:26.977090 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:44:26.977074 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b77941fb12f20d963572ad3ae27e72beaa73075a9879fb3d8c660164b41149d\": container with ID starting with 0b77941fb12f20d963572ad3ae27e72beaa73075a9879fb3d8c660164b41149d not found: ID does not exist" containerID="0b77941fb12f20d963572ad3ae27e72beaa73075a9879fb3d8c660164b41149d" Sep 10 15:44:26.977131 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.977096 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b77941fb12f20d963572ad3ae27e72beaa73075a9879fb3d8c660164b41149d"} err="failed to get container status \"0b77941fb12f20d963572ad3ae27e72beaa73075a9879fb3d8c660164b41149d\": rpc error: code = NotFound desc = could not find container \"0b77941fb12f20d963572ad3ae27e72beaa73075a9879fb3d8c660164b41149d\": container with ID starting with 0b77941fb12f20d963572ad3ae27e72beaa73075a9879fb3d8c660164b41149d not found: ID does not exist" Sep 10 15:44:26.977131 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.977113 2576 scope.go:117] "RemoveContainer" containerID="b6eeedab9e833977e99d59492dc5b9ae2868ef098b9909a92b24a09147552982" Sep 10 15:44:26.977370 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:44:26.977349 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6eeedab9e833977e99d59492dc5b9ae2868ef098b9909a92b24a09147552982\": container with ID starting with b6eeedab9e833977e99d59492dc5b9ae2868ef098b9909a92b24a09147552982 not found: ID does not exist" containerID="b6eeedab9e833977e99d59492dc5b9ae2868ef098b9909a92b24a09147552982" Sep 10 15:44:26.977451 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.977375 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6eeedab9e833977e99d59492dc5b9ae2868ef098b9909a92b24a09147552982"} err="failed to get container status \"b6eeedab9e833977e99d59492dc5b9ae2868ef098b9909a92b24a09147552982\": rpc error: code = NotFound desc = could not find container \"b6eeedab9e833977e99d59492dc5b9ae2868ef098b9909a92b24a09147552982\": container with ID starting with b6eeedab9e833977e99d59492dc5b9ae2868ef098b9909a92b24a09147552982 not found: ID does not exist" Sep 10 15:44:26.977451 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.977390 2576 scope.go:117] "RemoveContainer" containerID="580c96036a76d764882779627b5d80e30a2fa6ef6ce760e88fb119b7d0c32a27" Sep 10 15:44:26.977621 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:44:26.977606 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"580c96036a76d764882779627b5d80e30a2fa6ef6ce760e88fb119b7d0c32a27\": container with ID starting with 580c96036a76d764882779627b5d80e30a2fa6ef6ce760e88fb119b7d0c32a27 not found: ID does not exist" containerID="580c96036a76d764882779627b5d80e30a2fa6ef6ce760e88fb119b7d0c32a27" Sep 10 15:44:26.977659 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.977622 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"580c96036a76d764882779627b5d80e30a2fa6ef6ce760e88fb119b7d0c32a27"} err="failed to get container status \"580c96036a76d764882779627b5d80e30a2fa6ef6ce760e88fb119b7d0c32a27\": rpc error: code = NotFound desc = could not find container \"580c96036a76d764882779627b5d80e30a2fa6ef6ce760e88fb119b7d0c32a27\": container with ID starting with 580c96036a76d764882779627b5d80e30a2fa6ef6ce760e88fb119b7d0c32a27 not found: ID does not exist" Sep 10 15:44:26.977659 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.977637 2576 scope.go:117] "RemoveContainer" containerID="b15acc6ccc1036a8a2b74ba3d138e14aee1f71934e316e09413110d507c99f37" Sep 10 15:44:26.977868 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:44:26.977853 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b15acc6ccc1036a8a2b74ba3d138e14aee1f71934e316e09413110d507c99f37\": container with ID starting with b15acc6ccc1036a8a2b74ba3d138e14aee1f71934e316e09413110d507c99f37 not found: ID does not exist" containerID="b15acc6ccc1036a8a2b74ba3d138e14aee1f71934e316e09413110d507c99f37" Sep 10 15:44:26.977921 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.977872 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b15acc6ccc1036a8a2b74ba3d138e14aee1f71934e316e09413110d507c99f37"} err="failed to get container status \"b15acc6ccc1036a8a2b74ba3d138e14aee1f71934e316e09413110d507c99f37\": rpc error: code = NotFound desc = could not find container \"b15acc6ccc1036a8a2b74ba3d138e14aee1f71934e316e09413110d507c99f37\": container with ID starting with b15acc6ccc1036a8a2b74ba3d138e14aee1f71934e316e09413110d507c99f37 not found: ID does not exist" Sep 10 15:44:26.977921 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.977884 2576 scope.go:117] "RemoveContainer" containerID="2bebb7dcd3bb8af8304a2d5b718d5a84a065932fddb60ce095460bb246dce756" Sep 10 15:44:26.978118 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:44:26.978100 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bebb7dcd3bb8af8304a2d5b718d5a84a065932fddb60ce095460bb246dce756\": container with ID starting with 2bebb7dcd3bb8af8304a2d5b718d5a84a065932fddb60ce095460bb246dce756 not found: ID does not exist" containerID="2bebb7dcd3bb8af8304a2d5b718d5a84a065932fddb60ce095460bb246dce756" Sep 10 15:44:26.978202 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:26.978121 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bebb7dcd3bb8af8304a2d5b718d5a84a065932fddb60ce095460bb246dce756"} err="failed to get container status \"2bebb7dcd3bb8af8304a2d5b718d5a84a065932fddb60ce095460bb246dce756\": rpc error: code = NotFound desc = could not find container \"2bebb7dcd3bb8af8304a2d5b718d5a84a065932fddb60ce095460bb246dce756\": container with ID starting with 2bebb7dcd3bb8af8304a2d5b718d5a84a065932fddb60ce095460bb246dce756 not found: ID does not exist" Sep 10 15:44:27.168143 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:27.168117 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9dd9c02-cf57-49d8-a55c-0139a15c7eeb" path="/var/lib/kubelet/pods/d9dd9c02-cf57-49d8-a55c-0139a15c7eeb/volumes" Sep 10 15:44:29.090177 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.090136 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5b66c47b87-khlqk"] Sep 10 15:44:29.090660 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.090535 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9dd9c02-cf57-49d8-a55c-0139a15c7eeb" containerName="alertmanager" Sep 10 15:44:29.090660 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.090561 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9dd9c02-cf57-49d8-a55c-0139a15c7eeb" containerName="alertmanager" Sep 10 15:44:29.090660 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.090576 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af48efd8-d70a-4bc3-9f4f-250ce098c8e6" containerName="console" Sep 10 15:44:29.090660 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.090585 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="af48efd8-d70a-4bc3-9f4f-250ce098c8e6" containerName="console" Sep 10 15:44:29.090660 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.090594 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9dd9c02-cf57-49d8-a55c-0139a15c7eeb" containerName="kube-rbac-proxy-web" Sep 10 15:44:29.090660 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.090603 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9dd9c02-cf57-49d8-a55c-0139a15c7eeb" containerName="kube-rbac-proxy-web" Sep 10 15:44:29.090660 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.090623 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9dd9c02-cf57-49d8-a55c-0139a15c7eeb" containerName="prom-label-proxy" Sep 10 15:44:29.090660 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.090631 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9dd9c02-cf57-49d8-a55c-0139a15c7eeb" containerName="prom-label-proxy" Sep 10 15:44:29.090660 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.090641 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9dd9c02-cf57-49d8-a55c-0139a15c7eeb" containerName="kube-rbac-proxy" Sep 10 15:44:29.090660 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.090648 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9dd9c02-cf57-49d8-a55c-0139a15c7eeb" containerName="kube-rbac-proxy" Sep 10 15:44:29.090660 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.090659 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9dd9c02-cf57-49d8-a55c-0139a15c7eeb" containerName="kube-rbac-proxy-metric" Sep 10 15:44:29.090660 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.090667 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9dd9c02-cf57-49d8-a55c-0139a15c7eeb" containerName="kube-rbac-proxy-metric" Sep 10 15:44:29.091269 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.090679 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9dd9c02-cf57-49d8-a55c-0139a15c7eeb" containerName="config-reloader" Sep 10 15:44:29.091269 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.090687 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9dd9c02-cf57-49d8-a55c-0139a15c7eeb" containerName="config-reloader" Sep 10 15:44:29.091269 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.090702 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9dd9c02-cf57-49d8-a55c-0139a15c7eeb" containerName="init-config-reloader" Sep 10 15:44:29.091269 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.090711 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9dd9c02-cf57-49d8-a55c-0139a15c7eeb" containerName="init-config-reloader" Sep 10 15:44:29.091269 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.090804 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9dd9c02-cf57-49d8-a55c-0139a15c7eeb" containerName="config-reloader" Sep 10 15:44:29.091269 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.090816 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="af48efd8-d70a-4bc3-9f4f-250ce098c8e6" containerName="console" Sep 10 15:44:29.091269 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.090824 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9dd9c02-cf57-49d8-a55c-0139a15c7eeb" containerName="kube-rbac-proxy" Sep 10 15:44:29.091269 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.090832 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9dd9c02-cf57-49d8-a55c-0139a15c7eeb" containerName="kube-rbac-proxy-metric" Sep 10 15:44:29.091269 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.090839 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9dd9c02-cf57-49d8-a55c-0139a15c7eeb" containerName="prom-label-proxy" Sep 10 15:44:29.091269 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.090850 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9dd9c02-cf57-49d8-a55c-0139a15c7eeb" containerName="kube-rbac-proxy-web" Sep 10 15:44:29.091269 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.090858 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9dd9c02-cf57-49d8-a55c-0139a15c7eeb" containerName="alertmanager" Sep 10 15:44:29.095809 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.095788 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5b66c47b87-khlqk" Sep 10 15:44:29.098159 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.098137 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 10 15:44:29.098159 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.098137 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-9m4g5\"" Sep 10 15:44:29.098424 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.098407 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 10 15:44:29.098480 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.098458 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 10 15:44:29.098558 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.098407 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 10 15:44:29.098924 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.098908 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 10 15:44:29.104853 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.104831 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 10 15:44:29.105600 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.105577 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5b66c47b87-khlqk"] Sep 10 15:44:29.237201 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.237158 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9aa86db-f046-49a4-bba7-3a9ca2bdae16-serving-certs-ca-bundle\") pod \"telemeter-client-5b66c47b87-khlqk\" (UID: \"a9aa86db-f046-49a4-bba7-3a9ca2bdae16\") " pod="openshift-monitoring/telemeter-client-5b66c47b87-khlqk" Sep 10 15:44:29.237201 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.237201 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a9aa86db-f046-49a4-bba7-3a9ca2bdae16-federate-client-tls\") pod \"telemeter-client-5b66c47b87-khlqk\" (UID: \"a9aa86db-f046-49a4-bba7-3a9ca2bdae16\") " pod="openshift-monitoring/telemeter-client-5b66c47b87-khlqk" Sep 10 15:44:29.237444 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.237231 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9aa86db-f046-49a4-bba7-3a9ca2bdae16-metrics-client-ca\") pod \"telemeter-client-5b66c47b87-khlqk\" (UID: \"a9aa86db-f046-49a4-bba7-3a9ca2bdae16\") " pod="openshift-monitoring/telemeter-client-5b66c47b87-khlqk" Sep 10 15:44:29.237444 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.237329 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrk5p\" (UniqueName: \"kubernetes.io/projected/a9aa86db-f046-49a4-bba7-3a9ca2bdae16-kube-api-access-xrk5p\") pod \"telemeter-client-5b66c47b87-khlqk\" (UID: \"a9aa86db-f046-49a4-bba7-3a9ca2bdae16\") " pod="openshift-monitoring/telemeter-client-5b66c47b87-khlqk" Sep 10 15:44:29.237555 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.237460 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a9aa86db-f046-49a4-bba7-3a9ca2bdae16-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5b66c47b87-khlqk\" (UID: \"a9aa86db-f046-49a4-bba7-3a9ca2bdae16\") " pod="openshift-monitoring/telemeter-client-5b66c47b87-khlqk" Sep 10 15:44:29.237555 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.237491 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a9aa86db-f046-49a4-bba7-3a9ca2bdae16-secret-telemeter-client\") pod \"telemeter-client-5b66c47b87-khlqk\" (UID: \"a9aa86db-f046-49a4-bba7-3a9ca2bdae16\") " pod="openshift-monitoring/telemeter-client-5b66c47b87-khlqk" Sep 10 15:44:29.237555 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.237529 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9aa86db-f046-49a4-bba7-3a9ca2bdae16-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5b66c47b87-khlqk\" (UID: \"a9aa86db-f046-49a4-bba7-3a9ca2bdae16\") " pod="openshift-monitoring/telemeter-client-5b66c47b87-khlqk" Sep 10 15:44:29.237555 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.237551 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a9aa86db-f046-49a4-bba7-3a9ca2bdae16-telemeter-client-tls\") pod \"telemeter-client-5b66c47b87-khlqk\" (UID: \"a9aa86db-f046-49a4-bba7-3a9ca2bdae16\") " pod="openshift-monitoring/telemeter-client-5b66c47b87-khlqk" Sep 10 15:44:29.338922 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.338884 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a9aa86db-f046-49a4-bba7-3a9ca2bdae16-secret-telemeter-client\") pod \"telemeter-client-5b66c47b87-khlqk\" (UID: \"a9aa86db-f046-49a4-bba7-3a9ca2bdae16\") " pod="openshift-monitoring/telemeter-client-5b66c47b87-khlqk" Sep 10 15:44:29.338922 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.338933 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9aa86db-f046-49a4-bba7-3a9ca2bdae16-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5b66c47b87-khlqk\" (UID: \"a9aa86db-f046-49a4-bba7-3a9ca2bdae16\") " pod="openshift-monitoring/telemeter-client-5b66c47b87-khlqk" Sep 10 15:44:29.339196 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.338953 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a9aa86db-f046-49a4-bba7-3a9ca2bdae16-telemeter-client-tls\") pod \"telemeter-client-5b66c47b87-khlqk\" (UID: \"a9aa86db-f046-49a4-bba7-3a9ca2bdae16\") " pod="openshift-monitoring/telemeter-client-5b66c47b87-khlqk" Sep 10 15:44:29.339196 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.338983 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9aa86db-f046-49a4-bba7-3a9ca2bdae16-serving-certs-ca-bundle\") pod \"telemeter-client-5b66c47b87-khlqk\" (UID: \"a9aa86db-f046-49a4-bba7-3a9ca2bdae16\") " pod="openshift-monitoring/telemeter-client-5b66c47b87-khlqk" Sep 10 15:44:29.339196 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.338998 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a9aa86db-f046-49a4-bba7-3a9ca2bdae16-federate-client-tls\") pod \"telemeter-client-5b66c47b87-khlqk\" (UID: \"a9aa86db-f046-49a4-bba7-3a9ca2bdae16\") " pod="openshift-monitoring/telemeter-client-5b66c47b87-khlqk" Sep 10 15:44:29.339196 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.339017 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9aa86db-f046-49a4-bba7-3a9ca2bdae16-metrics-client-ca\") pod \"telemeter-client-5b66c47b87-khlqk\" (UID: \"a9aa86db-f046-49a4-bba7-3a9ca2bdae16\") " pod="openshift-monitoring/telemeter-client-5b66c47b87-khlqk" Sep 10 15:44:29.339196 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.339038 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrk5p\" (UniqueName: \"kubernetes.io/projected/a9aa86db-f046-49a4-bba7-3a9ca2bdae16-kube-api-access-xrk5p\") pod \"telemeter-client-5b66c47b87-khlqk\" (UID: \"a9aa86db-f046-49a4-bba7-3a9ca2bdae16\") " pod="openshift-monitoring/telemeter-client-5b66c47b87-khlqk" Sep 10 15:44:29.339472 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.339446 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a9aa86db-f046-49a4-bba7-3a9ca2bdae16-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5b66c47b87-khlqk\" (UID: \"a9aa86db-f046-49a4-bba7-3a9ca2bdae16\") " pod="openshift-monitoring/telemeter-client-5b66c47b87-khlqk" Sep 10 15:44:29.339911 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.339822 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9aa86db-f046-49a4-bba7-3a9ca2bdae16-serving-certs-ca-bundle\") pod \"telemeter-client-5b66c47b87-khlqk\" (UID: \"a9aa86db-f046-49a4-bba7-3a9ca2bdae16\") " pod="openshift-monitoring/telemeter-client-5b66c47b87-khlqk" Sep 10 15:44:29.339911 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.339868 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9aa86db-f046-49a4-bba7-3a9ca2bdae16-metrics-client-ca\") pod \"telemeter-client-5b66c47b87-khlqk\" (UID: \"a9aa86db-f046-49a4-bba7-3a9ca2bdae16\") " pod="openshift-monitoring/telemeter-client-5b66c47b87-khlqk" Sep 10 15:44:29.339911 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.339888 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9aa86db-f046-49a4-bba7-3a9ca2bdae16-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5b66c47b87-khlqk\" (UID: \"a9aa86db-f046-49a4-bba7-3a9ca2bdae16\") " pod="openshift-monitoring/telemeter-client-5b66c47b87-khlqk" Sep 10 15:44:29.341971 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.341912 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a9aa86db-f046-49a4-bba7-3a9ca2bdae16-telemeter-client-tls\") pod \"telemeter-client-5b66c47b87-khlqk\" (UID: \"a9aa86db-f046-49a4-bba7-3a9ca2bdae16\") " pod="openshift-monitoring/telemeter-client-5b66c47b87-khlqk" Sep 10 15:44:29.342064 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.342007 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a9aa86db-f046-49a4-bba7-3a9ca2bdae16-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5b66c47b87-khlqk\" (UID: \"a9aa86db-f046-49a4-bba7-3a9ca2bdae16\") " pod="openshift-monitoring/telemeter-client-5b66c47b87-khlqk" Sep 10 15:44:29.342216 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.342194 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a9aa86db-f046-49a4-bba7-3a9ca2bdae16-secret-telemeter-client\") pod \"telemeter-client-5b66c47b87-khlqk\" (UID: \"a9aa86db-f046-49a4-bba7-3a9ca2bdae16\") " pod="openshift-monitoring/telemeter-client-5b66c47b87-khlqk" Sep 10 15:44:29.342283 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.342260 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a9aa86db-f046-49a4-bba7-3a9ca2bdae16-federate-client-tls\") pod \"telemeter-client-5b66c47b87-khlqk\" (UID: \"a9aa86db-f046-49a4-bba7-3a9ca2bdae16\") " pod="openshift-monitoring/telemeter-client-5b66c47b87-khlqk" Sep 10 15:44:29.346531 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.346506 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrk5p\" (UniqueName: \"kubernetes.io/projected/a9aa86db-f046-49a4-bba7-3a9ca2bdae16-kube-api-access-xrk5p\") pod \"telemeter-client-5b66c47b87-khlqk\" (UID: \"a9aa86db-f046-49a4-bba7-3a9ca2bdae16\") " pod="openshift-monitoring/telemeter-client-5b66c47b87-khlqk" Sep 10 15:44:29.388236 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.388199 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 15:44:29.388522 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.388491 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b8e44c28-d988-4f54-8e09-363cf3cf55a2" containerName="prometheus" containerID="cri-o://418ce35d1d2fcf9b9456e40180323d30ba760be7d12e5b8c72d41ccaed4d646b" gracePeriod=600 Sep 10 15:44:29.388522 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.388512 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b8e44c28-d988-4f54-8e09-363cf3cf55a2" containerName="kube-rbac-proxy-web" containerID="cri-o://9a1afa6f48a3cba3f2adfda5f597365bb7c650932bb809e96e612ae9929bbdb4" gracePeriod=600 Sep 10 15:44:29.388634 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.388518 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b8e44c28-d988-4f54-8e09-363cf3cf55a2" containerName="thanos-sidecar" containerID="cri-o://45c12b3a07236dd8d6ab3511e23ec70dc18940a6176c72924e85cea49f2a5d95" gracePeriod=600 Sep 10 15:44:29.388634 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.388535 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b8e44c28-d988-4f54-8e09-363cf3cf55a2" containerName="kube-rbac-proxy-thanos" containerID="cri-o://5ce4293653053039cc902f4a45944b9c3c71c680f7fc5d47d80d8f57e8a11ffb" gracePeriod=600 Sep 10 15:44:29.388634 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.388561 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b8e44c28-d988-4f54-8e09-363cf3cf55a2" containerName="kube-rbac-proxy" containerID="cri-o://c9fbed8a5161c1472f48249a5be47597df0e77e8d6e2085faa723eebdcececf1" gracePeriod=600 Sep 10 15:44:29.388634 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.388523 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b8e44c28-d988-4f54-8e09-363cf3cf55a2" containerName="config-reloader" containerID="cri-o://d1924b4ff00fda092ad255cdbabcce72bac71ace4a3460c4790268e73ee7876f" gracePeriod=600 Sep 10 15:44:29.406353 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.406327 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5b66c47b87-khlqk" Sep 10 15:44:29.631613 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.631586 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:29.687536 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.687504 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5b66c47b87-khlqk"] Sep 10 15:44:29.690367 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:44:29.690344 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9aa86db_f046_49a4_bba7_3a9ca2bdae16.slice/crio-fdf398858f931c092a2bfcee05598c58712a23c4dbc93c79f2f67f468c39241e WatchSource:0}: Error finding container fdf398858f931c092a2bfcee05598c58712a23c4dbc93c79f2f67f468c39241e: Status 404 returned error can't find the container with id fdf398858f931c092a2bfcee05598c58712a23c4dbc93c79f2f67f468c39241e Sep 10 15:44:29.743122 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.743090 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b8e44c28-d988-4f54-8e09-363cf3cf55a2-web-config\") pod \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " Sep 10 15:44:29.743122 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.743120 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b8e44c28-d988-4f54-8e09-363cf3cf55a2-config\") pod \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " Sep 10 15:44:29.743357 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.743141 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b8e44c28-d988-4f54-8e09-363cf3cf55a2-thanos-prometheus-http-client-file\") pod \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " Sep 10 15:44:29.743357 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.743165 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8e44c28-d988-4f54-8e09-363cf3cf55a2-configmap-kubelet-serving-ca-bundle\") pod \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " Sep 10 15:44:29.743357 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.743197 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8e44c28-d988-4f54-8e09-363cf3cf55a2-configmap-metrics-client-ca\") pod \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " Sep 10 15:44:29.743357 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.743221 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8e44c28-d988-4f54-8e09-363cf3cf55a2-prometheus-trusted-ca-bundle\") pod \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " Sep 10 15:44:29.743357 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.743242 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b8e44c28-d988-4f54-8e09-363cf3cf55a2-secret-grpc-tls\") pod \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " Sep 10 15:44:29.743357 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.743283 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b8e44c28-d988-4f54-8e09-363cf3cf55a2-secret-kube-rbac-proxy\") pod \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " Sep 10 15:44:29.743357 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.743307 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b8e44c28-d988-4f54-8e09-363cf3cf55a2-tls-assets\") pod \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " Sep 10 15:44:29.743357 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.743338 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b8e44c28-d988-4f54-8e09-363cf3cf55a2-prometheus-k8s-db\") pod \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " Sep 10 15:44:29.743357 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.743359 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b8e44c28-d988-4f54-8e09-363cf3cf55a2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " Sep 10 15:44:29.743958 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.743387 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rpmgg\" (UniqueName: \"kubernetes.io/projected/b8e44c28-d988-4f54-8e09-363cf3cf55a2-kube-api-access-rpmgg\") pod \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " Sep 10 15:44:29.743958 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.743421 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b8e44c28-d988-4f54-8e09-363cf3cf55a2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " Sep 10 15:44:29.743958 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.743452 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8e44c28-d988-4f54-8e09-363cf3cf55a2-configmap-serving-certs-ca-bundle\") pod \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " Sep 10 15:44:29.743958 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.743493 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b8e44c28-d988-4f54-8e09-363cf3cf55a2-prometheus-k8s-rulefiles-1\") pod \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " Sep 10 15:44:29.743958 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.743528 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b8e44c28-d988-4f54-8e09-363cf3cf55a2-secret-prometheus-k8s-tls\") pod \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " Sep 10 15:44:29.743958 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.743555 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8e44c28-d988-4f54-8e09-363cf3cf55a2-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "b8e44c28-d988-4f54-8e09-363cf3cf55a2" (UID: "b8e44c28-d988-4f54-8e09-363cf3cf55a2"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:44:29.743958 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.743583 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b8e44c28-d988-4f54-8e09-363cf3cf55a2-prometheus-k8s-rulefiles-2\") pod \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " Sep 10 15:44:29.743958 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.743612 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b8e44c28-d988-4f54-8e09-363cf3cf55a2-secret-metrics-client-certs\") pod \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " Sep 10 15:44:29.743958 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.743638 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b8e44c28-d988-4f54-8e09-363cf3cf55a2-prometheus-k8s-rulefiles-0\") pod \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " Sep 10 15:44:29.743958 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.743673 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b8e44c28-d988-4f54-8e09-363cf3cf55a2-config-out\") pod \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\" (UID: \"b8e44c28-d988-4f54-8e09-363cf3cf55a2\") " Sep 10 15:44:29.744446 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.743887 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8e44c28-d988-4f54-8e09-363cf3cf55a2-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "b8e44c28-d988-4f54-8e09-363cf3cf55a2" (UID: "b8e44c28-d988-4f54-8e09-363cf3cf55a2"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:44:29.744446 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.743962 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8e44c28-d988-4f54-8e09-363cf3cf55a2-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "b8e44c28-d988-4f54-8e09-363cf3cf55a2" (UID: "b8e44c28-d988-4f54-8e09-363cf3cf55a2"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:44:29.744446 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.744095 2576 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8e44c28-d988-4f54-8e09-363cf3cf55a2-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:44:29.744446 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.744117 2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8e44c28-d988-4f54-8e09-363cf3cf55a2-prometheus-trusted-ca-bundle\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:44:29.744446 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.744135 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8e44c28-d988-4f54-8e09-363cf3cf55a2-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "b8e44c28-d988-4f54-8e09-363cf3cf55a2" (UID: "b8e44c28-d988-4f54-8e09-363cf3cf55a2"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:44:29.744446 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.744167 2576 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8e44c28-d988-4f54-8e09-363cf3cf55a2-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:44:29.746079 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.746050 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8e44c28-d988-4f54-8e09-363cf3cf55a2-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "b8e44c28-d988-4f54-8e09-363cf3cf55a2" (UID: "b8e44c28-d988-4f54-8e09-363cf3cf55a2"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:44:29.746191 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.746120 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8e44c28-d988-4f54-8e09-363cf3cf55a2-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "b8e44c28-d988-4f54-8e09-363cf3cf55a2" (UID: "b8e44c28-d988-4f54-8e09-363cf3cf55a2"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:44:29.746191 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.746144 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8e44c28-d988-4f54-8e09-363cf3cf55a2-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "b8e44c28-d988-4f54-8e09-363cf3cf55a2" (UID: "b8e44c28-d988-4f54-8e09-363cf3cf55a2"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:44:29.746314 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.746252 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8e44c28-d988-4f54-8e09-363cf3cf55a2-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "b8e44c28-d988-4f54-8e09-363cf3cf55a2" (UID: "b8e44c28-d988-4f54-8e09-363cf3cf55a2"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:44:29.746576 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.746555 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8e44c28-d988-4f54-8e09-363cf3cf55a2-config" (OuterVolumeSpecName: "config") pod "b8e44c28-d988-4f54-8e09-363cf3cf55a2" (UID: "b8e44c28-d988-4f54-8e09-363cf3cf55a2"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:44:29.746649 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.746631 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8e44c28-d988-4f54-8e09-363cf3cf55a2-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "b8e44c28-d988-4f54-8e09-363cf3cf55a2" (UID: "b8e44c28-d988-4f54-8e09-363cf3cf55a2"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:44:29.746920 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.746889 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8e44c28-d988-4f54-8e09-363cf3cf55a2-config-out" (OuterVolumeSpecName: "config-out") pod "b8e44c28-d988-4f54-8e09-363cf3cf55a2" (UID: "b8e44c28-d988-4f54-8e09-363cf3cf55a2"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:44:29.747003 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.746965 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8e44c28-d988-4f54-8e09-363cf3cf55a2-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "b8e44c28-d988-4f54-8e09-363cf3cf55a2" (UID: "b8e44c28-d988-4f54-8e09-363cf3cf55a2"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:44:29.748069 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.748007 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8e44c28-d988-4f54-8e09-363cf3cf55a2-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "b8e44c28-d988-4f54-8e09-363cf3cf55a2" (UID: "b8e44c28-d988-4f54-8e09-363cf3cf55a2"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:44:29.748539 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.748508 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8e44c28-d988-4f54-8e09-363cf3cf55a2-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "b8e44c28-d988-4f54-8e09-363cf3cf55a2" (UID: "b8e44c28-d988-4f54-8e09-363cf3cf55a2"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:44:29.748811 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.748775 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8e44c28-d988-4f54-8e09-363cf3cf55a2-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "b8e44c28-d988-4f54-8e09-363cf3cf55a2" (UID: "b8e44c28-d988-4f54-8e09-363cf3cf55a2"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:44:29.748945 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.748890 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8e44c28-d988-4f54-8e09-363cf3cf55a2-kube-api-access-rpmgg" (OuterVolumeSpecName: "kube-api-access-rpmgg") pod "b8e44c28-d988-4f54-8e09-363cf3cf55a2" (UID: "b8e44c28-d988-4f54-8e09-363cf3cf55a2"). InnerVolumeSpecName "kube-api-access-rpmgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:44:29.748945 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.748927 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8e44c28-d988-4f54-8e09-363cf3cf55a2-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "b8e44c28-d988-4f54-8e09-363cf3cf55a2" (UID: "b8e44c28-d988-4f54-8e09-363cf3cf55a2"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:44:29.749067 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.748967 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8e44c28-d988-4f54-8e09-363cf3cf55a2-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "b8e44c28-d988-4f54-8e09-363cf3cf55a2" (UID: "b8e44c28-d988-4f54-8e09-363cf3cf55a2"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:44:29.749335 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.749320 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8e44c28-d988-4f54-8e09-363cf3cf55a2-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "b8e44c28-d988-4f54-8e09-363cf3cf55a2" (UID: "b8e44c28-d988-4f54-8e09-363cf3cf55a2"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:44:29.757343 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.757319 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8e44c28-d988-4f54-8e09-363cf3cf55a2-web-config" (OuterVolumeSpecName: "web-config") pod "b8e44c28-d988-4f54-8e09-363cf3cf55a2" (UID: "b8e44c28-d988-4f54-8e09-363cf3cf55a2"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:44:29.844885 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.844852 2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b8e44c28-d988-4f54-8e09-363cf3cf55a2-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:44:29.844885 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.844878 2576 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b8e44c28-d988-4f54-8e09-363cf3cf55a2-config-out\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:44:29.844885 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.844887 2576 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b8e44c28-d988-4f54-8e09-363cf3cf55a2-web-config\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:44:29.845098 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.844897 2576 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/b8e44c28-d988-4f54-8e09-363cf3cf55a2-config\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:44:29.845098 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.844906 2576 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b8e44c28-d988-4f54-8e09-363cf3cf55a2-thanos-prometheus-http-client-file\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:44:29.845098 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.844914 2576 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8e44c28-d988-4f54-8e09-363cf3cf55a2-configmap-metrics-client-ca\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:44:29.845098 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.844923 2576 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b8e44c28-d988-4f54-8e09-363cf3cf55a2-secret-grpc-tls\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:44:29.845098 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.844930 2576 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b8e44c28-d988-4f54-8e09-363cf3cf55a2-secret-kube-rbac-proxy\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:44:29.845098 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.844938 2576 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b8e44c28-d988-4f54-8e09-363cf3cf55a2-tls-assets\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:44:29.845098 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.844946 2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b8e44c28-d988-4f54-8e09-363cf3cf55a2-prometheus-k8s-db\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:44:29.845098 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.844954 2576 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b8e44c28-d988-4f54-8e09-363cf3cf55a2-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:44:29.845098 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.844963 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpmgg\" (UniqueName: \"kubernetes.io/projected/b8e44c28-d988-4f54-8e09-363cf3cf55a2-kube-api-access-rpmgg\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:44:29.845098 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.844971 2576 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b8e44c28-d988-4f54-8e09-363cf3cf55a2-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:44:29.845098 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.844980 2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b8e44c28-d988-4f54-8e09-363cf3cf55a2-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:44:29.845098 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.844988 2576 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b8e44c28-d988-4f54-8e09-363cf3cf55a2-secret-prometheus-k8s-tls\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:44:29.845098 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.844995 2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b8e44c28-d988-4f54-8e09-363cf3cf55a2-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:44:29.845098 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.845003 2576 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b8e44c28-d988-4f54-8e09-363cf3cf55a2-secret-metrics-client-certs\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:44:29.941457 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.941379 2576 generic.go:356] "Generic (PLEG): container finished" podID="b8e44c28-d988-4f54-8e09-363cf3cf55a2" containerID="5ce4293653053039cc902f4a45944b9c3c71c680f7fc5d47d80d8f57e8a11ffb" exitCode=0 Sep 10 15:44:29.941457 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.941402 2576 generic.go:356] "Generic (PLEG): container finished" podID="b8e44c28-d988-4f54-8e09-363cf3cf55a2" containerID="c9fbed8a5161c1472f48249a5be47597df0e77e8d6e2085faa723eebdcececf1" exitCode=0 Sep 10 15:44:29.941457 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.941409 2576 generic.go:356] "Generic (PLEG): container finished" podID="b8e44c28-d988-4f54-8e09-363cf3cf55a2" containerID="9a1afa6f48a3cba3f2adfda5f597365bb7c650932bb809e96e612ae9929bbdb4" exitCode=0 Sep 10 15:44:29.941457 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.941415 2576 generic.go:356] "Generic (PLEG): container finished" podID="b8e44c28-d988-4f54-8e09-363cf3cf55a2" containerID="45c12b3a07236dd8d6ab3511e23ec70dc18940a6176c72924e85cea49f2a5d95" exitCode=0 Sep 10 15:44:29.941457 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.941420 2576 generic.go:356] "Generic (PLEG): container finished" podID="b8e44c28-d988-4f54-8e09-363cf3cf55a2" containerID="d1924b4ff00fda092ad255cdbabcce72bac71ace4a3460c4790268e73ee7876f" exitCode=0 Sep 10 15:44:29.941457 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.941425 2576 generic.go:356] "Generic (PLEG): container finished" podID="b8e44c28-d988-4f54-8e09-363cf3cf55a2" containerID="418ce35d1d2fcf9b9456e40180323d30ba760be7d12e5b8c72d41ccaed4d646b" exitCode=0 Sep 10 15:44:29.941860 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.941470 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:29.941860 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.941469 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8e44c28-d988-4f54-8e09-363cf3cf55a2","Type":"ContainerDied","Data":"5ce4293653053039cc902f4a45944b9c3c71c680f7fc5d47d80d8f57e8a11ffb"} Sep 10 15:44:29.941860 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.941628 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8e44c28-d988-4f54-8e09-363cf3cf55a2","Type":"ContainerDied","Data":"c9fbed8a5161c1472f48249a5be47597df0e77e8d6e2085faa723eebdcececf1"} Sep 10 15:44:29.941860 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.941663 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8e44c28-d988-4f54-8e09-363cf3cf55a2","Type":"ContainerDied","Data":"9a1afa6f48a3cba3f2adfda5f597365bb7c650932bb809e96e612ae9929bbdb4"} Sep 10 15:44:29.941860 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.941692 2576 scope.go:117] "RemoveContainer" containerID="5ce4293653053039cc902f4a45944b9c3c71c680f7fc5d47d80d8f57e8a11ffb" Sep 10 15:44:29.941860 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.941701 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8e44c28-d988-4f54-8e09-363cf3cf55a2","Type":"ContainerDied","Data":"45c12b3a07236dd8d6ab3511e23ec70dc18940a6176c72924e85cea49f2a5d95"} Sep 10 15:44:29.941860 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.941770 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8e44c28-d988-4f54-8e09-363cf3cf55a2","Type":"ContainerDied","Data":"d1924b4ff00fda092ad255cdbabcce72bac71ace4a3460c4790268e73ee7876f"} Sep 10 15:44:29.941860 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.941798 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8e44c28-d988-4f54-8e09-363cf3cf55a2","Type":"ContainerDied","Data":"418ce35d1d2fcf9b9456e40180323d30ba760be7d12e5b8c72d41ccaed4d646b"} Sep 10 15:44:29.941860 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.941826 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8e44c28-d988-4f54-8e09-363cf3cf55a2","Type":"ContainerDied","Data":"d11e9c43b1d5209fc04670646a978c72c1db04a457d3cb6d98dc89a59ac09a0d"} Sep 10 15:44:29.943031 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.942906 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5b66c47b87-khlqk" event={"ID":"a9aa86db-f046-49a4-bba7-3a9ca2bdae16","Type":"ContainerStarted","Data":"fdf398858f931c092a2bfcee05598c58712a23c4dbc93c79f2f67f468c39241e"} Sep 10 15:44:29.949793 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.949778 2576 scope.go:117] "RemoveContainer" containerID="c9fbed8a5161c1472f48249a5be47597df0e77e8d6e2085faa723eebdcececf1" Sep 10 15:44:29.956457 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.956442 2576 scope.go:117] "RemoveContainer" containerID="9a1afa6f48a3cba3f2adfda5f597365bb7c650932bb809e96e612ae9929bbdb4" Sep 10 15:44:29.962568 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.962545 2576 scope.go:117] "RemoveContainer" containerID="45c12b3a07236dd8d6ab3511e23ec70dc18940a6176c72924e85cea49f2a5d95" Sep 10 15:44:29.965296 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.965275 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 15:44:29.969727 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.969708 2576 scope.go:117] "RemoveContainer" containerID="d1924b4ff00fda092ad255cdbabcce72bac71ace4a3460c4790268e73ee7876f" Sep 10 15:44:29.970408 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.970387 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 15:44:29.976113 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.976095 2576 scope.go:117] "RemoveContainer" containerID="418ce35d1d2fcf9b9456e40180323d30ba760be7d12e5b8c72d41ccaed4d646b" Sep 10 15:44:29.982835 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.982817 2576 scope.go:117] "RemoveContainer" containerID="11df3c11441b3cecabfedb1c89b5d6d61ab7351dbab4bab6b5e898a13abe0de4" Sep 10 15:44:29.989526 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.989510 2576 scope.go:117] "RemoveContainer" containerID="5ce4293653053039cc902f4a45944b9c3c71c680f7fc5d47d80d8f57e8a11ffb" Sep 10 15:44:29.989900 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:44:29.989879 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ce4293653053039cc902f4a45944b9c3c71c680f7fc5d47d80d8f57e8a11ffb\": container with ID starting with 5ce4293653053039cc902f4a45944b9c3c71c680f7fc5d47d80d8f57e8a11ffb not found: ID does not exist" containerID="5ce4293653053039cc902f4a45944b9c3c71c680f7fc5d47d80d8f57e8a11ffb" Sep 10 15:44:29.989939 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.989907 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ce4293653053039cc902f4a45944b9c3c71c680f7fc5d47d80d8f57e8a11ffb"} err="failed to get container status \"5ce4293653053039cc902f4a45944b9c3c71c680f7fc5d47d80d8f57e8a11ffb\": rpc error: code = NotFound desc = could not find container \"5ce4293653053039cc902f4a45944b9c3c71c680f7fc5d47d80d8f57e8a11ffb\": container with ID starting with 5ce4293653053039cc902f4a45944b9c3c71c680f7fc5d47d80d8f57e8a11ffb not found: ID does not exist" Sep 10 15:44:29.989939 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.989925 2576 scope.go:117] "RemoveContainer" containerID="c9fbed8a5161c1472f48249a5be47597df0e77e8d6e2085faa723eebdcececf1" Sep 10 15:44:29.990162 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:44:29.990145 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9fbed8a5161c1472f48249a5be47597df0e77e8d6e2085faa723eebdcececf1\": container with ID starting with c9fbed8a5161c1472f48249a5be47597df0e77e8d6e2085faa723eebdcececf1 not found: ID does not exist" containerID="c9fbed8a5161c1472f48249a5be47597df0e77e8d6e2085faa723eebdcececf1" Sep 10 15:44:29.990207 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.990169 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9fbed8a5161c1472f48249a5be47597df0e77e8d6e2085faa723eebdcececf1"} err="failed to get container status \"c9fbed8a5161c1472f48249a5be47597df0e77e8d6e2085faa723eebdcececf1\": rpc error: code = NotFound desc = could not find container \"c9fbed8a5161c1472f48249a5be47597df0e77e8d6e2085faa723eebdcececf1\": container with ID starting with c9fbed8a5161c1472f48249a5be47597df0e77e8d6e2085faa723eebdcececf1 not found: ID does not exist" Sep 10 15:44:29.990207 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.990184 2576 scope.go:117] "RemoveContainer" containerID="9a1afa6f48a3cba3f2adfda5f597365bb7c650932bb809e96e612ae9929bbdb4" Sep 10 15:44:29.990409 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:44:29.990393 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a1afa6f48a3cba3f2adfda5f597365bb7c650932bb809e96e612ae9929bbdb4\": container with ID starting with 9a1afa6f48a3cba3f2adfda5f597365bb7c650932bb809e96e612ae9929bbdb4 not found: ID does not exist" containerID="9a1afa6f48a3cba3f2adfda5f597365bb7c650932bb809e96e612ae9929bbdb4" Sep 10 15:44:29.990451 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.990413 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a1afa6f48a3cba3f2adfda5f597365bb7c650932bb809e96e612ae9929bbdb4"} err="failed to get container status \"9a1afa6f48a3cba3f2adfda5f597365bb7c650932bb809e96e612ae9929bbdb4\": rpc error: code = NotFound desc = could not find container \"9a1afa6f48a3cba3f2adfda5f597365bb7c650932bb809e96e612ae9929bbdb4\": container with ID starting with 9a1afa6f48a3cba3f2adfda5f597365bb7c650932bb809e96e612ae9929bbdb4 not found: ID does not exist" Sep 10 15:44:29.990451 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.990426 2576 scope.go:117] "RemoveContainer" containerID="45c12b3a07236dd8d6ab3511e23ec70dc18940a6176c72924e85cea49f2a5d95" Sep 10 15:44:29.990673 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:44:29.990658 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45c12b3a07236dd8d6ab3511e23ec70dc18940a6176c72924e85cea49f2a5d95\": container with ID starting with 45c12b3a07236dd8d6ab3511e23ec70dc18940a6176c72924e85cea49f2a5d95 not found: ID does not exist" containerID="45c12b3a07236dd8d6ab3511e23ec70dc18940a6176c72924e85cea49f2a5d95" Sep 10 15:44:29.990714 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.990675 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45c12b3a07236dd8d6ab3511e23ec70dc18940a6176c72924e85cea49f2a5d95"} err="failed to get container status \"45c12b3a07236dd8d6ab3511e23ec70dc18940a6176c72924e85cea49f2a5d95\": rpc error: code = NotFound desc = could not find container \"45c12b3a07236dd8d6ab3511e23ec70dc18940a6176c72924e85cea49f2a5d95\": container with ID starting with 45c12b3a07236dd8d6ab3511e23ec70dc18940a6176c72924e85cea49f2a5d95 not found: ID does not exist" Sep 10 15:44:29.990714 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.990687 2576 scope.go:117] "RemoveContainer" containerID="d1924b4ff00fda092ad255cdbabcce72bac71ace4a3460c4790268e73ee7876f" Sep 10 15:44:29.990902 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:44:29.990888 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1924b4ff00fda092ad255cdbabcce72bac71ace4a3460c4790268e73ee7876f\": container with ID starting with d1924b4ff00fda092ad255cdbabcce72bac71ace4a3460c4790268e73ee7876f not found: ID does not exist" containerID="d1924b4ff00fda092ad255cdbabcce72bac71ace4a3460c4790268e73ee7876f" Sep 10 15:44:29.990948 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.990904 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1924b4ff00fda092ad255cdbabcce72bac71ace4a3460c4790268e73ee7876f"} err="failed to get container status \"d1924b4ff00fda092ad255cdbabcce72bac71ace4a3460c4790268e73ee7876f\": rpc error: code = NotFound desc = could not find container \"d1924b4ff00fda092ad255cdbabcce72bac71ace4a3460c4790268e73ee7876f\": container with ID starting with d1924b4ff00fda092ad255cdbabcce72bac71ace4a3460c4790268e73ee7876f not found: ID does not exist" Sep 10 15:44:29.990948 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.990916 2576 scope.go:117] "RemoveContainer" containerID="418ce35d1d2fcf9b9456e40180323d30ba760be7d12e5b8c72d41ccaed4d646b" Sep 10 15:44:29.991108 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:44:29.991094 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"418ce35d1d2fcf9b9456e40180323d30ba760be7d12e5b8c72d41ccaed4d646b\": container with ID starting with 418ce35d1d2fcf9b9456e40180323d30ba760be7d12e5b8c72d41ccaed4d646b not found: ID does not exist" containerID="418ce35d1d2fcf9b9456e40180323d30ba760be7d12e5b8c72d41ccaed4d646b" Sep 10 15:44:29.991146 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.991110 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"418ce35d1d2fcf9b9456e40180323d30ba760be7d12e5b8c72d41ccaed4d646b"} err="failed to get container status \"418ce35d1d2fcf9b9456e40180323d30ba760be7d12e5b8c72d41ccaed4d646b\": rpc error: code = NotFound desc = could not find container \"418ce35d1d2fcf9b9456e40180323d30ba760be7d12e5b8c72d41ccaed4d646b\": container with ID starting with 418ce35d1d2fcf9b9456e40180323d30ba760be7d12e5b8c72d41ccaed4d646b not found: ID does not exist" Sep 10 15:44:29.991146 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.991121 2576 scope.go:117] "RemoveContainer" containerID="11df3c11441b3cecabfedb1c89b5d6d61ab7351dbab4bab6b5e898a13abe0de4" Sep 10 15:44:29.991347 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:44:29.991326 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11df3c11441b3cecabfedb1c89b5d6d61ab7351dbab4bab6b5e898a13abe0de4\": container with ID starting with 11df3c11441b3cecabfedb1c89b5d6d61ab7351dbab4bab6b5e898a13abe0de4 not found: ID does not exist" containerID="11df3c11441b3cecabfedb1c89b5d6d61ab7351dbab4bab6b5e898a13abe0de4" Sep 10 15:44:29.991384 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.991351 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11df3c11441b3cecabfedb1c89b5d6d61ab7351dbab4bab6b5e898a13abe0de4"} err="failed to get container status \"11df3c11441b3cecabfedb1c89b5d6d61ab7351dbab4bab6b5e898a13abe0de4\": rpc error: code = NotFound desc = could not find container \"11df3c11441b3cecabfedb1c89b5d6d61ab7351dbab4bab6b5e898a13abe0de4\": container with ID starting with 11df3c11441b3cecabfedb1c89b5d6d61ab7351dbab4bab6b5e898a13abe0de4 not found: ID does not exist" Sep 10 15:44:29.991384 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.991362 2576 scope.go:117] "RemoveContainer" containerID="5ce4293653053039cc902f4a45944b9c3c71c680f7fc5d47d80d8f57e8a11ffb" Sep 10 15:44:29.991596 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.991568 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ce4293653053039cc902f4a45944b9c3c71c680f7fc5d47d80d8f57e8a11ffb"} err="failed to get container status \"5ce4293653053039cc902f4a45944b9c3c71c680f7fc5d47d80d8f57e8a11ffb\": rpc error: code = NotFound desc = could not find container \"5ce4293653053039cc902f4a45944b9c3c71c680f7fc5d47d80d8f57e8a11ffb\": container with ID starting with 5ce4293653053039cc902f4a45944b9c3c71c680f7fc5d47d80d8f57e8a11ffb not found: ID does not exist" Sep 10 15:44:29.991638 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.991598 2576 scope.go:117] "RemoveContainer" containerID="c9fbed8a5161c1472f48249a5be47597df0e77e8d6e2085faa723eebdcececf1" Sep 10 15:44:29.991902 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.991884 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9fbed8a5161c1472f48249a5be47597df0e77e8d6e2085faa723eebdcececf1"} err="failed to get container status \"c9fbed8a5161c1472f48249a5be47597df0e77e8d6e2085faa723eebdcececf1\": rpc error: code = NotFound desc = could not find container \"c9fbed8a5161c1472f48249a5be47597df0e77e8d6e2085faa723eebdcececf1\": container with ID starting with c9fbed8a5161c1472f48249a5be47597df0e77e8d6e2085faa723eebdcececf1 not found: ID does not exist" Sep 10 15:44:29.991971 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.991902 2576 scope.go:117] "RemoveContainer" containerID="9a1afa6f48a3cba3f2adfda5f597365bb7c650932bb809e96e612ae9929bbdb4" Sep 10 15:44:29.992118 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.992101 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a1afa6f48a3cba3f2adfda5f597365bb7c650932bb809e96e612ae9929bbdb4"} err="failed to get container status \"9a1afa6f48a3cba3f2adfda5f597365bb7c650932bb809e96e612ae9929bbdb4\": rpc error: code = NotFound desc = could not find container \"9a1afa6f48a3cba3f2adfda5f597365bb7c650932bb809e96e612ae9929bbdb4\": container with ID starting with 9a1afa6f48a3cba3f2adfda5f597365bb7c650932bb809e96e612ae9929bbdb4 not found: ID does not exist" Sep 10 15:44:29.992159 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.992117 2576 scope.go:117] "RemoveContainer" containerID="45c12b3a07236dd8d6ab3511e23ec70dc18940a6176c72924e85cea49f2a5d95" Sep 10 15:44:29.992342 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.992327 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45c12b3a07236dd8d6ab3511e23ec70dc18940a6176c72924e85cea49f2a5d95"} err="failed to get container status \"45c12b3a07236dd8d6ab3511e23ec70dc18940a6176c72924e85cea49f2a5d95\": rpc error: code = NotFound desc = could not find container \"45c12b3a07236dd8d6ab3511e23ec70dc18940a6176c72924e85cea49f2a5d95\": container with ID starting with 45c12b3a07236dd8d6ab3511e23ec70dc18940a6176c72924e85cea49f2a5d95 not found: ID does not exist" Sep 10 15:44:29.992391 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.992343 2576 scope.go:117] "RemoveContainer" containerID="d1924b4ff00fda092ad255cdbabcce72bac71ace4a3460c4790268e73ee7876f" Sep 10 15:44:29.992516 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.992501 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1924b4ff00fda092ad255cdbabcce72bac71ace4a3460c4790268e73ee7876f"} err="failed to get container status \"d1924b4ff00fda092ad255cdbabcce72bac71ace4a3460c4790268e73ee7876f\": rpc error: code = NotFound desc = could not find container \"d1924b4ff00fda092ad255cdbabcce72bac71ace4a3460c4790268e73ee7876f\": container with ID starting with d1924b4ff00fda092ad255cdbabcce72bac71ace4a3460c4790268e73ee7876f not found: ID does not exist" Sep 10 15:44:29.992559 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.992516 2576 scope.go:117] "RemoveContainer" containerID="418ce35d1d2fcf9b9456e40180323d30ba760be7d12e5b8c72d41ccaed4d646b" Sep 10 15:44:29.992685 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.992667 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"418ce35d1d2fcf9b9456e40180323d30ba760be7d12e5b8c72d41ccaed4d646b"} err="failed to get container status \"418ce35d1d2fcf9b9456e40180323d30ba760be7d12e5b8c72d41ccaed4d646b\": rpc error: code = NotFound desc = could not find container \"418ce35d1d2fcf9b9456e40180323d30ba760be7d12e5b8c72d41ccaed4d646b\": container with ID starting with 418ce35d1d2fcf9b9456e40180323d30ba760be7d12e5b8c72d41ccaed4d646b not found: ID does not exist" Sep 10 15:44:29.992721 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.992686 2576 scope.go:117] "RemoveContainer" containerID="11df3c11441b3cecabfedb1c89b5d6d61ab7351dbab4bab6b5e898a13abe0de4" Sep 10 15:44:29.992882 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.992863 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11df3c11441b3cecabfedb1c89b5d6d61ab7351dbab4bab6b5e898a13abe0de4"} err="failed to get container status \"11df3c11441b3cecabfedb1c89b5d6d61ab7351dbab4bab6b5e898a13abe0de4\": rpc error: code = NotFound desc = could not find container \"11df3c11441b3cecabfedb1c89b5d6d61ab7351dbab4bab6b5e898a13abe0de4\": container with ID starting with 11df3c11441b3cecabfedb1c89b5d6d61ab7351dbab4bab6b5e898a13abe0de4 not found: ID does not exist" Sep 10 15:44:29.992979 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.992881 2576 scope.go:117] "RemoveContainer" containerID="5ce4293653053039cc902f4a45944b9c3c71c680f7fc5d47d80d8f57e8a11ffb" Sep 10 15:44:29.993122 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.993107 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ce4293653053039cc902f4a45944b9c3c71c680f7fc5d47d80d8f57e8a11ffb"} err="failed to get container status \"5ce4293653053039cc902f4a45944b9c3c71c680f7fc5d47d80d8f57e8a11ffb\": rpc error: code = NotFound desc = could not find container \"5ce4293653053039cc902f4a45944b9c3c71c680f7fc5d47d80d8f57e8a11ffb\": container with ID starting with 5ce4293653053039cc902f4a45944b9c3c71c680f7fc5d47d80d8f57e8a11ffb not found: ID does not exist" Sep 10 15:44:29.993162 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.993131 2576 scope.go:117] "RemoveContainer" containerID="c9fbed8a5161c1472f48249a5be47597df0e77e8d6e2085faa723eebdcececf1" Sep 10 15:44:29.993385 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.993367 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9fbed8a5161c1472f48249a5be47597df0e77e8d6e2085faa723eebdcececf1"} err="failed to get container status \"c9fbed8a5161c1472f48249a5be47597df0e77e8d6e2085faa723eebdcececf1\": rpc error: code = NotFound desc = could not find container \"c9fbed8a5161c1472f48249a5be47597df0e77e8d6e2085faa723eebdcececf1\": container with ID starting with c9fbed8a5161c1472f48249a5be47597df0e77e8d6e2085faa723eebdcececf1 not found: ID does not exist" Sep 10 15:44:29.993431 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.993387 2576 scope.go:117] "RemoveContainer" containerID="9a1afa6f48a3cba3f2adfda5f597365bb7c650932bb809e96e612ae9929bbdb4" Sep 10 15:44:29.993613 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.993596 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a1afa6f48a3cba3f2adfda5f597365bb7c650932bb809e96e612ae9929bbdb4"} err="failed to get container status \"9a1afa6f48a3cba3f2adfda5f597365bb7c650932bb809e96e612ae9929bbdb4\": rpc error: code = NotFound desc = could not find container \"9a1afa6f48a3cba3f2adfda5f597365bb7c650932bb809e96e612ae9929bbdb4\": container with ID starting with 9a1afa6f48a3cba3f2adfda5f597365bb7c650932bb809e96e612ae9929bbdb4 not found: ID does not exist" Sep 10 15:44:29.993657 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.993613 2576 scope.go:117] "RemoveContainer" containerID="45c12b3a07236dd8d6ab3511e23ec70dc18940a6176c72924e85cea49f2a5d95" Sep 10 15:44:29.993872 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.993854 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45c12b3a07236dd8d6ab3511e23ec70dc18940a6176c72924e85cea49f2a5d95"} err="failed to get container status \"45c12b3a07236dd8d6ab3511e23ec70dc18940a6176c72924e85cea49f2a5d95\": rpc error: code = NotFound desc = could not find container \"45c12b3a07236dd8d6ab3511e23ec70dc18940a6176c72924e85cea49f2a5d95\": container with ID starting with 45c12b3a07236dd8d6ab3511e23ec70dc18940a6176c72924e85cea49f2a5d95 not found: ID does not exist" Sep 10 15:44:29.993951 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.993875 2576 scope.go:117] "RemoveContainer" containerID="d1924b4ff00fda092ad255cdbabcce72bac71ace4a3460c4790268e73ee7876f" Sep 10 15:44:29.994132 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.994116 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1924b4ff00fda092ad255cdbabcce72bac71ace4a3460c4790268e73ee7876f"} err="failed to get container status \"d1924b4ff00fda092ad255cdbabcce72bac71ace4a3460c4790268e73ee7876f\": rpc error: code = NotFound desc = could not find container \"d1924b4ff00fda092ad255cdbabcce72bac71ace4a3460c4790268e73ee7876f\": container with ID starting with d1924b4ff00fda092ad255cdbabcce72bac71ace4a3460c4790268e73ee7876f not found: ID does not exist" Sep 10 15:44:29.994173 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.994133 2576 scope.go:117] "RemoveContainer" containerID="418ce35d1d2fcf9b9456e40180323d30ba760be7d12e5b8c72d41ccaed4d646b" Sep 10 15:44:29.994359 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.994342 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"418ce35d1d2fcf9b9456e40180323d30ba760be7d12e5b8c72d41ccaed4d646b"} err="failed to get container status \"418ce35d1d2fcf9b9456e40180323d30ba760be7d12e5b8c72d41ccaed4d646b\": rpc error: code = NotFound desc = could not find container \"418ce35d1d2fcf9b9456e40180323d30ba760be7d12e5b8c72d41ccaed4d646b\": container with ID starting with 418ce35d1d2fcf9b9456e40180323d30ba760be7d12e5b8c72d41ccaed4d646b not found: ID does not exist" Sep 10 15:44:29.994409 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.994359 2576 scope.go:117] "RemoveContainer" containerID="11df3c11441b3cecabfedb1c89b5d6d61ab7351dbab4bab6b5e898a13abe0de4" Sep 10 15:44:29.994565 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.994540 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11df3c11441b3cecabfedb1c89b5d6d61ab7351dbab4bab6b5e898a13abe0de4"} err="failed to get container status \"11df3c11441b3cecabfedb1c89b5d6d61ab7351dbab4bab6b5e898a13abe0de4\": rpc error: code = NotFound desc = could not find container \"11df3c11441b3cecabfedb1c89b5d6d61ab7351dbab4bab6b5e898a13abe0de4\": container with ID starting with 11df3c11441b3cecabfedb1c89b5d6d61ab7351dbab4bab6b5e898a13abe0de4 not found: ID does not exist" Sep 10 15:44:29.994633 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.994568 2576 scope.go:117] "RemoveContainer" containerID="5ce4293653053039cc902f4a45944b9c3c71c680f7fc5d47d80d8f57e8a11ffb" Sep 10 15:44:29.994893 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.994866 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ce4293653053039cc902f4a45944b9c3c71c680f7fc5d47d80d8f57e8a11ffb"} err="failed to get container status \"5ce4293653053039cc902f4a45944b9c3c71c680f7fc5d47d80d8f57e8a11ffb\": rpc error: code = NotFound desc = could not find container \"5ce4293653053039cc902f4a45944b9c3c71c680f7fc5d47d80d8f57e8a11ffb\": container with ID starting with 5ce4293653053039cc902f4a45944b9c3c71c680f7fc5d47d80d8f57e8a11ffb not found: ID does not exist" Sep 10 15:44:29.994978 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.994894 2576 scope.go:117] "RemoveContainer" containerID="c9fbed8a5161c1472f48249a5be47597df0e77e8d6e2085faa723eebdcececf1" Sep 10 15:44:29.995273 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.995239 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9fbed8a5161c1472f48249a5be47597df0e77e8d6e2085faa723eebdcececf1"} err="failed to get container status \"c9fbed8a5161c1472f48249a5be47597df0e77e8d6e2085faa723eebdcececf1\": rpc error: code = NotFound desc = could not find container \"c9fbed8a5161c1472f48249a5be47597df0e77e8d6e2085faa723eebdcececf1\": container with ID starting with c9fbed8a5161c1472f48249a5be47597df0e77e8d6e2085faa723eebdcececf1 not found: ID does not exist" Sep 10 15:44:29.995273 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.995265 2576 scope.go:117] "RemoveContainer" containerID="9a1afa6f48a3cba3f2adfda5f597365bb7c650932bb809e96e612ae9929bbdb4" Sep 10 15:44:29.995948 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.995654 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a1afa6f48a3cba3f2adfda5f597365bb7c650932bb809e96e612ae9929bbdb4"} err="failed to get container status \"9a1afa6f48a3cba3f2adfda5f597365bb7c650932bb809e96e612ae9929bbdb4\": rpc error: code = NotFound desc = could not find container \"9a1afa6f48a3cba3f2adfda5f597365bb7c650932bb809e96e612ae9929bbdb4\": container with ID starting with 9a1afa6f48a3cba3f2adfda5f597365bb7c650932bb809e96e612ae9929bbdb4 not found: ID does not exist" Sep 10 15:44:29.995948 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.995680 2576 scope.go:117] "RemoveContainer" containerID="45c12b3a07236dd8d6ab3511e23ec70dc18940a6176c72924e85cea49f2a5d95" Sep 10 15:44:29.996255 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.996234 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45c12b3a07236dd8d6ab3511e23ec70dc18940a6176c72924e85cea49f2a5d95"} err="failed to get container status \"45c12b3a07236dd8d6ab3511e23ec70dc18940a6176c72924e85cea49f2a5d95\": rpc error: code = NotFound desc = could not find container \"45c12b3a07236dd8d6ab3511e23ec70dc18940a6176c72924e85cea49f2a5d95\": container with ID starting with 45c12b3a07236dd8d6ab3511e23ec70dc18940a6176c72924e85cea49f2a5d95 not found: ID does not exist" Sep 10 15:44:29.996331 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.996258 2576 scope.go:117] "RemoveContainer" containerID="d1924b4ff00fda092ad255cdbabcce72bac71ace4a3460c4790268e73ee7876f" Sep 10 15:44:29.996529 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.996508 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1924b4ff00fda092ad255cdbabcce72bac71ace4a3460c4790268e73ee7876f"} err="failed to get container status \"d1924b4ff00fda092ad255cdbabcce72bac71ace4a3460c4790268e73ee7876f\": rpc error: code = NotFound desc = could not find container \"d1924b4ff00fda092ad255cdbabcce72bac71ace4a3460c4790268e73ee7876f\": container with ID starting with d1924b4ff00fda092ad255cdbabcce72bac71ace4a3460c4790268e73ee7876f not found: ID does not exist" Sep 10 15:44:29.996585 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.996531 2576 scope.go:117] "RemoveContainer" containerID="418ce35d1d2fcf9b9456e40180323d30ba760be7d12e5b8c72d41ccaed4d646b" Sep 10 15:44:29.996744 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.996713 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"418ce35d1d2fcf9b9456e40180323d30ba760be7d12e5b8c72d41ccaed4d646b"} err="failed to get container status \"418ce35d1d2fcf9b9456e40180323d30ba760be7d12e5b8c72d41ccaed4d646b\": rpc error: code = NotFound desc = could not find container \"418ce35d1d2fcf9b9456e40180323d30ba760be7d12e5b8c72d41ccaed4d646b\": container with ID starting with 418ce35d1d2fcf9b9456e40180323d30ba760be7d12e5b8c72d41ccaed4d646b not found: ID does not exist" Sep 10 15:44:29.996797 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.996751 2576 scope.go:117] "RemoveContainer" containerID="11df3c11441b3cecabfedb1c89b5d6d61ab7351dbab4bab6b5e898a13abe0de4" Sep 10 15:44:29.996797 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.996772 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 15:44:29.996986 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.996965 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11df3c11441b3cecabfedb1c89b5d6d61ab7351dbab4bab6b5e898a13abe0de4"} err="failed to get container status \"11df3c11441b3cecabfedb1c89b5d6d61ab7351dbab4bab6b5e898a13abe0de4\": rpc error: code = NotFound desc = could not find container \"11df3c11441b3cecabfedb1c89b5d6d61ab7351dbab4bab6b5e898a13abe0de4\": container with ID starting with 11df3c11441b3cecabfedb1c89b5d6d61ab7351dbab4bab6b5e898a13abe0de4 not found: ID does not exist" Sep 10 15:44:29.996986 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.996984 2576 scope.go:117] "RemoveContainer" containerID="5ce4293653053039cc902f4a45944b9c3c71c680f7fc5d47d80d8f57e8a11ffb" Sep 10 15:44:29.997112 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.997095 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8e44c28-d988-4f54-8e09-363cf3cf55a2" containerName="kube-rbac-proxy" Sep 10 15:44:29.997153 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.997114 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8e44c28-d988-4f54-8e09-363cf3cf55a2" containerName="kube-rbac-proxy" Sep 10 15:44:29.997153 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.997131 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8e44c28-d988-4f54-8e09-363cf3cf55a2" containerName="kube-rbac-proxy-web" Sep 10 15:44:29.997153 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.997140 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8e44c28-d988-4f54-8e09-363cf3cf55a2" containerName="kube-rbac-proxy-web" Sep 10 15:44:29.997153 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.997151 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8e44c28-d988-4f54-8e09-363cf3cf55a2" containerName="init-config-reloader" Sep 10 15:44:29.997288 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.997157 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8e44c28-d988-4f54-8e09-363cf3cf55a2" containerName="init-config-reloader" Sep 10 15:44:29.997288 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.997163 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8e44c28-d988-4f54-8e09-363cf3cf55a2" containerName="kube-rbac-proxy-thanos" Sep 10 15:44:29.997288 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.997167 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8e44c28-d988-4f54-8e09-363cf3cf55a2" containerName="kube-rbac-proxy-thanos" Sep 10 15:44:29.997288 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.997175 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8e44c28-d988-4f54-8e09-363cf3cf55a2" containerName="prometheus" Sep 10 15:44:29.997288 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.997180 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8e44c28-d988-4f54-8e09-363cf3cf55a2" containerName="prometheus" Sep 10 15:44:29.997288 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.997187 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8e44c28-d988-4f54-8e09-363cf3cf55a2" containerName="thanos-sidecar" Sep 10 15:44:29.997288 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.997191 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8e44c28-d988-4f54-8e09-363cf3cf55a2" containerName="thanos-sidecar" Sep 10 15:44:29.997288 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.997201 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8e44c28-d988-4f54-8e09-363cf3cf55a2" containerName="config-reloader" Sep 10 15:44:29.997288 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.997205 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8e44c28-d988-4f54-8e09-363cf3cf55a2" containerName="config-reloader" Sep 10 15:44:29.997288 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.997203 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ce4293653053039cc902f4a45944b9c3c71c680f7fc5d47d80d8f57e8a11ffb"} err="failed to get container status \"5ce4293653053039cc902f4a45944b9c3c71c680f7fc5d47d80d8f57e8a11ffb\": rpc error: code = NotFound desc = could not find container \"5ce4293653053039cc902f4a45944b9c3c71c680f7fc5d47d80d8f57e8a11ffb\": container with ID starting with 5ce4293653053039cc902f4a45944b9c3c71c680f7fc5d47d80d8f57e8a11ffb not found: ID does not exist" Sep 10 15:44:29.997288 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.997223 2576 scope.go:117] "RemoveContainer" containerID="c9fbed8a5161c1472f48249a5be47597df0e77e8d6e2085faa723eebdcececf1" Sep 10 15:44:29.997288 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.997254 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8e44c28-d988-4f54-8e09-363cf3cf55a2" containerName="prometheus" Sep 10 15:44:29.997288 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.997262 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8e44c28-d988-4f54-8e09-363cf3cf55a2" containerName="config-reloader" Sep 10 15:44:29.997288 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.997267 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8e44c28-d988-4f54-8e09-363cf3cf55a2" containerName="thanos-sidecar" Sep 10 15:44:29.997288 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.997272 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8e44c28-d988-4f54-8e09-363cf3cf55a2" containerName="kube-rbac-proxy-thanos" Sep 10 15:44:29.997288 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.997277 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8e44c28-d988-4f54-8e09-363cf3cf55a2" containerName="kube-rbac-proxy" Sep 10 15:44:29.997288 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.997283 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8e44c28-d988-4f54-8e09-363cf3cf55a2" containerName="kube-rbac-proxy-web" Sep 10 15:44:29.997868 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.997444 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9fbed8a5161c1472f48249a5be47597df0e77e8d6e2085faa723eebdcececf1"} err="failed to get container status \"c9fbed8a5161c1472f48249a5be47597df0e77e8d6e2085faa723eebdcececf1\": rpc error: code = NotFound desc = could not find container \"c9fbed8a5161c1472f48249a5be47597df0e77e8d6e2085faa723eebdcececf1\": container with ID starting with c9fbed8a5161c1472f48249a5be47597df0e77e8d6e2085faa723eebdcececf1 not found: ID does not exist" Sep 10 15:44:29.997868 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.997462 2576 scope.go:117] "RemoveContainer" containerID="9a1afa6f48a3cba3f2adfda5f597365bb7c650932bb809e96e612ae9929bbdb4" Sep 10 15:44:29.997868 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.997646 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a1afa6f48a3cba3f2adfda5f597365bb7c650932bb809e96e612ae9929bbdb4"} err="failed to get container status \"9a1afa6f48a3cba3f2adfda5f597365bb7c650932bb809e96e612ae9929bbdb4\": rpc error: code = NotFound desc = could not find container \"9a1afa6f48a3cba3f2adfda5f597365bb7c650932bb809e96e612ae9929bbdb4\": container with ID starting with 9a1afa6f48a3cba3f2adfda5f597365bb7c650932bb809e96e612ae9929bbdb4 not found: ID does not exist" Sep 10 15:44:29.997868 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.997660 2576 scope.go:117] "RemoveContainer" containerID="45c12b3a07236dd8d6ab3511e23ec70dc18940a6176c72924e85cea49f2a5d95" Sep 10 15:44:29.998007 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.997886 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45c12b3a07236dd8d6ab3511e23ec70dc18940a6176c72924e85cea49f2a5d95"} err="failed to get container status \"45c12b3a07236dd8d6ab3511e23ec70dc18940a6176c72924e85cea49f2a5d95\": rpc error: code = NotFound desc = could not find container \"45c12b3a07236dd8d6ab3511e23ec70dc18940a6176c72924e85cea49f2a5d95\": container with ID starting with 45c12b3a07236dd8d6ab3511e23ec70dc18940a6176c72924e85cea49f2a5d95 not found: ID does not exist" Sep 10 15:44:29.998007 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.997897 2576 scope.go:117] "RemoveContainer" containerID="d1924b4ff00fda092ad255cdbabcce72bac71ace4a3460c4790268e73ee7876f" Sep 10 15:44:29.998166 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.998151 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1924b4ff00fda092ad255cdbabcce72bac71ace4a3460c4790268e73ee7876f"} err="failed to get container status \"d1924b4ff00fda092ad255cdbabcce72bac71ace4a3460c4790268e73ee7876f\": rpc error: code = NotFound desc = could not find container \"d1924b4ff00fda092ad255cdbabcce72bac71ace4a3460c4790268e73ee7876f\": container with ID starting with d1924b4ff00fda092ad255cdbabcce72bac71ace4a3460c4790268e73ee7876f not found: ID does not exist" Sep 10 15:44:29.998215 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.998166 2576 scope.go:117] "RemoveContainer" containerID="418ce35d1d2fcf9b9456e40180323d30ba760be7d12e5b8c72d41ccaed4d646b" Sep 10 15:44:29.998387 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.998372 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"418ce35d1d2fcf9b9456e40180323d30ba760be7d12e5b8c72d41ccaed4d646b"} err="failed to get container status \"418ce35d1d2fcf9b9456e40180323d30ba760be7d12e5b8c72d41ccaed4d646b\": rpc error: code = NotFound desc = could not find container \"418ce35d1d2fcf9b9456e40180323d30ba760be7d12e5b8c72d41ccaed4d646b\": container with ID starting with 418ce35d1d2fcf9b9456e40180323d30ba760be7d12e5b8c72d41ccaed4d646b not found: ID does not exist" Sep 10 15:44:29.998453 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.998387 2576 scope.go:117] "RemoveContainer" containerID="11df3c11441b3cecabfedb1c89b5d6d61ab7351dbab4bab6b5e898a13abe0de4" Sep 10 15:44:29.998609 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.998586 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11df3c11441b3cecabfedb1c89b5d6d61ab7351dbab4bab6b5e898a13abe0de4"} err="failed to get container status \"11df3c11441b3cecabfedb1c89b5d6d61ab7351dbab4bab6b5e898a13abe0de4\": rpc error: code = NotFound desc = could not find container \"11df3c11441b3cecabfedb1c89b5d6d61ab7351dbab4bab6b5e898a13abe0de4\": container with ID starting with 11df3c11441b3cecabfedb1c89b5d6d61ab7351dbab4bab6b5e898a13abe0de4 not found: ID does not exist" Sep 10 15:44:29.998609 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.998602 2576 scope.go:117] "RemoveContainer" containerID="5ce4293653053039cc902f4a45944b9c3c71c680f7fc5d47d80d8f57e8a11ffb" Sep 10 15:44:29.998852 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.998827 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ce4293653053039cc902f4a45944b9c3c71c680f7fc5d47d80d8f57e8a11ffb"} err="failed to get container status \"5ce4293653053039cc902f4a45944b9c3c71c680f7fc5d47d80d8f57e8a11ffb\": rpc error: code = NotFound desc = could not find container \"5ce4293653053039cc902f4a45944b9c3c71c680f7fc5d47d80d8f57e8a11ffb\": container with ID starting with 5ce4293653053039cc902f4a45944b9c3c71c680f7fc5d47d80d8f57e8a11ffb not found: ID does not exist" Sep 10 15:44:29.998927 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.998853 2576 scope.go:117] "RemoveContainer" containerID="c9fbed8a5161c1472f48249a5be47597df0e77e8d6e2085faa723eebdcececf1" Sep 10 15:44:29.999080 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.999062 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9fbed8a5161c1472f48249a5be47597df0e77e8d6e2085faa723eebdcececf1"} err="failed to get container status \"c9fbed8a5161c1472f48249a5be47597df0e77e8d6e2085faa723eebdcececf1\": rpc error: code = NotFound desc = could not find container \"c9fbed8a5161c1472f48249a5be47597df0e77e8d6e2085faa723eebdcececf1\": container with ID starting with c9fbed8a5161c1472f48249a5be47597df0e77e8d6e2085faa723eebdcececf1 not found: ID does not exist" Sep 10 15:44:29.999130 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.999081 2576 scope.go:117] "RemoveContainer" containerID="9a1afa6f48a3cba3f2adfda5f597365bb7c650932bb809e96e612ae9929bbdb4" Sep 10 15:44:29.999289 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.999272 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a1afa6f48a3cba3f2adfda5f597365bb7c650932bb809e96e612ae9929bbdb4"} err="failed to get container status \"9a1afa6f48a3cba3f2adfda5f597365bb7c650932bb809e96e612ae9929bbdb4\": rpc error: code = NotFound desc = could not find container \"9a1afa6f48a3cba3f2adfda5f597365bb7c650932bb809e96e612ae9929bbdb4\": container with ID starting with 9a1afa6f48a3cba3f2adfda5f597365bb7c650932bb809e96e612ae9929bbdb4 not found: ID does not exist" Sep 10 15:44:29.999351 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.999291 2576 scope.go:117] "RemoveContainer" containerID="45c12b3a07236dd8d6ab3511e23ec70dc18940a6176c72924e85cea49f2a5d95" Sep 10 15:44:29.999535 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.999517 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45c12b3a07236dd8d6ab3511e23ec70dc18940a6176c72924e85cea49f2a5d95"} err="failed to get container status \"45c12b3a07236dd8d6ab3511e23ec70dc18940a6176c72924e85cea49f2a5d95\": rpc error: code = NotFound desc = could not find container \"45c12b3a07236dd8d6ab3511e23ec70dc18940a6176c72924e85cea49f2a5d95\": container with ID starting with 45c12b3a07236dd8d6ab3511e23ec70dc18940a6176c72924e85cea49f2a5d95 not found: ID does not exist" Sep 10 15:44:29.999585 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.999536 2576 scope.go:117] "RemoveContainer" containerID="d1924b4ff00fda092ad255cdbabcce72bac71ace4a3460c4790268e73ee7876f" Sep 10 15:44:29.999781 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.999761 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1924b4ff00fda092ad255cdbabcce72bac71ace4a3460c4790268e73ee7876f"} err="failed to get container status \"d1924b4ff00fda092ad255cdbabcce72bac71ace4a3460c4790268e73ee7876f\": rpc error: code = NotFound desc = could not find container \"d1924b4ff00fda092ad255cdbabcce72bac71ace4a3460c4790268e73ee7876f\": container with ID starting with d1924b4ff00fda092ad255cdbabcce72bac71ace4a3460c4790268e73ee7876f not found: ID does not exist" Sep 10 15:44:29.999845 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:29.999782 2576 scope.go:117] "RemoveContainer" containerID="418ce35d1d2fcf9b9456e40180323d30ba760be7d12e5b8c72d41ccaed4d646b" Sep 10 15:44:30.000042 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.000025 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"418ce35d1d2fcf9b9456e40180323d30ba760be7d12e5b8c72d41ccaed4d646b"} err="failed to get container status \"418ce35d1d2fcf9b9456e40180323d30ba760be7d12e5b8c72d41ccaed4d646b\": rpc error: code = NotFound desc = could not find container \"418ce35d1d2fcf9b9456e40180323d30ba760be7d12e5b8c72d41ccaed4d646b\": container with ID starting with 418ce35d1d2fcf9b9456e40180323d30ba760be7d12e5b8c72d41ccaed4d646b not found: ID does not exist" Sep 10 15:44:30.000107 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.000042 2576 scope.go:117] "RemoveContainer" containerID="11df3c11441b3cecabfedb1c89b5d6d61ab7351dbab4bab6b5e898a13abe0de4" Sep 10 15:44:30.000322 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.000306 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11df3c11441b3cecabfedb1c89b5d6d61ab7351dbab4bab6b5e898a13abe0de4"} err="failed to get container status \"11df3c11441b3cecabfedb1c89b5d6d61ab7351dbab4bab6b5e898a13abe0de4\": rpc error: code = NotFound desc = could not find container \"11df3c11441b3cecabfedb1c89b5d6d61ab7351dbab4bab6b5e898a13abe0de4\": container with ID starting with 11df3c11441b3cecabfedb1c89b5d6d61ab7351dbab4bab6b5e898a13abe0de4 not found: ID does not exist" Sep 10 15:44:30.002514 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.002499 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:30.004921 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.004904 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 10 15:44:30.004993 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.004904 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-qhzpz\"" Sep 10 15:44:30.005112 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.005096 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 10 15:44:30.005250 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.005197 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 10 15:44:30.005359 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.005281 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 10 15:44:30.005359 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.005322 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 10 15:44:30.005478 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.005449 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 10 15:44:30.005527 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.005493 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 10 15:44:30.005592 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.005575 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 10 15:44:30.005694 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.005677 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 10 15:44:30.005776 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.005708 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 10 15:44:30.005828 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.005773 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 10 15:44:30.005918 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.005904 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-34cuaeejtnnt6\"" Sep 10 15:44:30.006268 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.006253 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 10 15:44:30.013412 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.011656 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 10 15:44:30.015653 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.015631 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 10 15:44:30.017768 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.017747 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 15:44:30.148099 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.148063 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91f948cc-e301-4b86-a1e8-a74ac8fba5ed-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"91f948cc-e301-4b86-a1e8-a74ac8fba5ed\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:30.148099 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.148099 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/91f948cc-e301-4b86-a1e8-a74ac8fba5ed-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"91f948cc-e301-4b86-a1e8-a74ac8fba5ed\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:30.148549 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.148127 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91f948cc-e301-4b86-a1e8-a74ac8fba5ed-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"91f948cc-e301-4b86-a1e8-a74ac8fba5ed\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:30.148549 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.148194 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/91f948cc-e301-4b86-a1e8-a74ac8fba5ed-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"91f948cc-e301-4b86-a1e8-a74ac8fba5ed\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:30.148549 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.148285 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/91f948cc-e301-4b86-a1e8-a74ac8fba5ed-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"91f948cc-e301-4b86-a1e8-a74ac8fba5ed\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:30.148549 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.148325 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/91f948cc-e301-4b86-a1e8-a74ac8fba5ed-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"91f948cc-e301-4b86-a1e8-a74ac8fba5ed\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:30.148549 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.148348 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/91f948cc-e301-4b86-a1e8-a74ac8fba5ed-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"91f948cc-e301-4b86-a1e8-a74ac8fba5ed\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:30.148549 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.148365 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/91f948cc-e301-4b86-a1e8-a74ac8fba5ed-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"91f948cc-e301-4b86-a1e8-a74ac8fba5ed\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:30.148549 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.148389 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91f948cc-e301-4b86-a1e8-a74ac8fba5ed-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"91f948cc-e301-4b86-a1e8-a74ac8fba5ed\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:30.148549 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.148404 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/91f948cc-e301-4b86-a1e8-a74ac8fba5ed-web-config\") pod \"prometheus-k8s-0\" (UID: \"91f948cc-e301-4b86-a1e8-a74ac8fba5ed\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:30.148549 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.148417 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91f948cc-e301-4b86-a1e8-a74ac8fba5ed-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"91f948cc-e301-4b86-a1e8-a74ac8fba5ed\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:30.148549 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.148434 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/91f948cc-e301-4b86-a1e8-a74ac8fba5ed-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"91f948cc-e301-4b86-a1e8-a74ac8fba5ed\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:30.148549 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.148451 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/91f948cc-e301-4b86-a1e8-a74ac8fba5ed-config\") pod \"prometheus-k8s-0\" (UID: \"91f948cc-e301-4b86-a1e8-a74ac8fba5ed\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:30.148549 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.148472 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/91f948cc-e301-4b86-a1e8-a74ac8fba5ed-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"91f948cc-e301-4b86-a1e8-a74ac8fba5ed\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:30.148549 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.148493 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/91f948cc-e301-4b86-a1e8-a74ac8fba5ed-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"91f948cc-e301-4b86-a1e8-a74ac8fba5ed\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:30.148549 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.148508 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/91f948cc-e301-4b86-a1e8-a74ac8fba5ed-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"91f948cc-e301-4b86-a1e8-a74ac8fba5ed\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:30.148549 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.148539 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/91f948cc-e301-4b86-a1e8-a74ac8fba5ed-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"91f948cc-e301-4b86-a1e8-a74ac8fba5ed\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:30.148549 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.148557 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhvfz\" (UniqueName: \"kubernetes.io/projected/91f948cc-e301-4b86-a1e8-a74ac8fba5ed-kube-api-access-xhvfz\") pod \"prometheus-k8s-0\" (UID: \"91f948cc-e301-4b86-a1e8-a74ac8fba5ed\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:30.149059 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.148574 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/91f948cc-e301-4b86-a1e8-a74ac8fba5ed-config-out\") pod \"prometheus-k8s-0\" (UID: \"91f948cc-e301-4b86-a1e8-a74ac8fba5ed\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:30.149059 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.148594 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/91f948cc-e301-4b86-a1e8-a74ac8fba5ed-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"91f948cc-e301-4b86-a1e8-a74ac8fba5ed\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:30.249117 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.249037 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/91f948cc-e301-4b86-a1e8-a74ac8fba5ed-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"91f948cc-e301-4b86-a1e8-a74ac8fba5ed\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:30.249117 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.249072 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/91f948cc-e301-4b86-a1e8-a74ac8fba5ed-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"91f948cc-e301-4b86-a1e8-a74ac8fba5ed\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:30.249117 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.249090 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/91f948cc-e301-4b86-a1e8-a74ac8fba5ed-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"91f948cc-e301-4b86-a1e8-a74ac8fba5ed\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:30.249117 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.249108 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91f948cc-e301-4b86-a1e8-a74ac8fba5ed-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"91f948cc-e301-4b86-a1e8-a74ac8fba5ed\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:30.249117 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.249123 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/91f948cc-e301-4b86-a1e8-a74ac8fba5ed-web-config\") pod \"prometheus-k8s-0\" (UID: \"91f948cc-e301-4b86-a1e8-a74ac8fba5ed\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:30.249446 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.249140 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91f948cc-e301-4b86-a1e8-a74ac8fba5ed-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"91f948cc-e301-4b86-a1e8-a74ac8fba5ed\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:30.249446 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.249165 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/91f948cc-e301-4b86-a1e8-a74ac8fba5ed-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"91f948cc-e301-4b86-a1e8-a74ac8fba5ed\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:30.249832 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.249677 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/91f948cc-e301-4b86-a1e8-a74ac8fba5ed-config\") pod \"prometheus-k8s-0\" (UID: \"91f948cc-e301-4b86-a1e8-a74ac8fba5ed\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:30.249832 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.249727 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/91f948cc-e301-4b86-a1e8-a74ac8fba5ed-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"91f948cc-e301-4b86-a1e8-a74ac8fba5ed\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:30.249832 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.249808 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/91f948cc-e301-4b86-a1e8-a74ac8fba5ed-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"91f948cc-e301-4b86-a1e8-a74ac8fba5ed\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:30.250042 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.249835 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/91f948cc-e301-4b86-a1e8-a74ac8fba5ed-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"91f948cc-e301-4b86-a1e8-a74ac8fba5ed\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:30.250042 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.249863 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/91f948cc-e301-4b86-a1e8-a74ac8fba5ed-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"91f948cc-e301-4b86-a1e8-a74ac8fba5ed\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:30.250042 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.249870 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/91f948cc-e301-4b86-a1e8-a74ac8fba5ed-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"91f948cc-e301-4b86-a1e8-a74ac8fba5ed\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:30.250042 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.249923 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhvfz\" (UniqueName: \"kubernetes.io/projected/91f948cc-e301-4b86-a1e8-a74ac8fba5ed-kube-api-access-xhvfz\") pod \"prometheus-k8s-0\" (UID: \"91f948cc-e301-4b86-a1e8-a74ac8fba5ed\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:30.250042 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.249955 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/91f948cc-e301-4b86-a1e8-a74ac8fba5ed-config-out\") pod \"prometheus-k8s-0\" (UID: \"91f948cc-e301-4b86-a1e8-a74ac8fba5ed\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:30.250042 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.249975 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91f948cc-e301-4b86-a1e8-a74ac8fba5ed-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"91f948cc-e301-4b86-a1e8-a74ac8fba5ed\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:30.250042 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.249982 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/91f948cc-e301-4b86-a1e8-a74ac8fba5ed-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"91f948cc-e301-4b86-a1e8-a74ac8fba5ed\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:30.250563 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.250045 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91f948cc-e301-4b86-a1e8-a74ac8fba5ed-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"91f948cc-e301-4b86-a1e8-a74ac8fba5ed\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:30.250563 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.250074 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/91f948cc-e301-4b86-a1e8-a74ac8fba5ed-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"91f948cc-e301-4b86-a1e8-a74ac8fba5ed\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:30.250563 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.250125 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91f948cc-e301-4b86-a1e8-a74ac8fba5ed-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"91f948cc-e301-4b86-a1e8-a74ac8fba5ed\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:30.250563 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.250162 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/91f948cc-e301-4b86-a1e8-a74ac8fba5ed-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"91f948cc-e301-4b86-a1e8-a74ac8fba5ed\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:30.250563 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.250200 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/91f948cc-e301-4b86-a1e8-a74ac8fba5ed-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"91f948cc-e301-4b86-a1e8-a74ac8fba5ed\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:30.250563 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.250314 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91f948cc-e301-4b86-a1e8-a74ac8fba5ed-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"91f948cc-e301-4b86-a1e8-a74ac8fba5ed\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:30.251002 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.250915 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/91f948cc-e301-4b86-a1e8-a74ac8fba5ed-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"91f948cc-e301-4b86-a1e8-a74ac8fba5ed\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:30.252716 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.251517 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91f948cc-e301-4b86-a1e8-a74ac8fba5ed-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"91f948cc-e301-4b86-a1e8-a74ac8fba5ed\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:30.252716 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.252315 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/91f948cc-e301-4b86-a1e8-a74ac8fba5ed-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"91f948cc-e301-4b86-a1e8-a74ac8fba5ed\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:30.252716 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.252366 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/91f948cc-e301-4b86-a1e8-a74ac8fba5ed-web-config\") pod \"prometheus-k8s-0\" (UID: \"91f948cc-e301-4b86-a1e8-a74ac8fba5ed\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:30.252716 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.252576 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/91f948cc-e301-4b86-a1e8-a74ac8fba5ed-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"91f948cc-e301-4b86-a1e8-a74ac8fba5ed\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:30.252716 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.252646 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/91f948cc-e301-4b86-a1e8-a74ac8fba5ed-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"91f948cc-e301-4b86-a1e8-a74ac8fba5ed\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:30.253239 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.252850 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/91f948cc-e301-4b86-a1e8-a74ac8fba5ed-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"91f948cc-e301-4b86-a1e8-a74ac8fba5ed\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:30.253239 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.253025 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91f948cc-e301-4b86-a1e8-a74ac8fba5ed-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"91f948cc-e301-4b86-a1e8-a74ac8fba5ed\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:30.253239 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.253077 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/91f948cc-e301-4b86-a1e8-a74ac8fba5ed-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"91f948cc-e301-4b86-a1e8-a74ac8fba5ed\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:30.253239 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.253138 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/91f948cc-e301-4b86-a1e8-a74ac8fba5ed-config-out\") pod \"prometheus-k8s-0\" (UID: \"91f948cc-e301-4b86-a1e8-a74ac8fba5ed\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:30.253591 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.253570 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/91f948cc-e301-4b86-a1e8-a74ac8fba5ed-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"91f948cc-e301-4b86-a1e8-a74ac8fba5ed\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:30.254242 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.254217 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/91f948cc-e301-4b86-a1e8-a74ac8fba5ed-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"91f948cc-e301-4b86-a1e8-a74ac8fba5ed\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:30.255150 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.255122 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/91f948cc-e301-4b86-a1e8-a74ac8fba5ed-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"91f948cc-e301-4b86-a1e8-a74ac8fba5ed\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:30.255253 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.255241 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/91f948cc-e301-4b86-a1e8-a74ac8fba5ed-config\") pod \"prometheus-k8s-0\" (UID: \"91f948cc-e301-4b86-a1e8-a74ac8fba5ed\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:30.255315 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.255271 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/91f948cc-e301-4b86-a1e8-a74ac8fba5ed-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"91f948cc-e301-4b86-a1e8-a74ac8fba5ed\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:30.256227 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.256206 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/91f948cc-e301-4b86-a1e8-a74ac8fba5ed-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"91f948cc-e301-4b86-a1e8-a74ac8fba5ed\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:30.259082 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.259065 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhvfz\" (UniqueName: \"kubernetes.io/projected/91f948cc-e301-4b86-a1e8-a74ac8fba5ed-kube-api-access-xhvfz\") pod \"prometheus-k8s-0\" (UID: \"91f948cc-e301-4b86-a1e8-a74ac8fba5ed\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:30.316149 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.316112 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:30.608297 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.607941 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 15:44:30.610494 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:44:30.610466 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91f948cc_e301_4b86_a1e8_a74ac8fba5ed.slice/crio-fc81aa11032b938f194de8a6b8264df6f85b09f507f55240c4089b53432fffe6 WatchSource:0}: Error finding container fc81aa11032b938f194de8a6b8264df6f85b09f507f55240c4089b53432fffe6: Status 404 returned error can't find the container with id fc81aa11032b938f194de8a6b8264df6f85b09f507f55240c4089b53432fffe6 Sep 10 15:44:30.948931 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.948846 2576 generic.go:356] "Generic (PLEG): container finished" podID="91f948cc-e301-4b86-a1e8-a74ac8fba5ed" containerID="7636e37eda1251ae0fd0b2ef48e249f1fffe6e7511cca3eb2229fc081c83ac6b" exitCode=0 Sep 10 15:44:30.948931 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.948907 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"91f948cc-e301-4b86-a1e8-a74ac8fba5ed","Type":"ContainerDied","Data":"7636e37eda1251ae0fd0b2ef48e249f1fffe6e7511cca3eb2229fc081c83ac6b"} Sep 10 15:44:30.949158 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:30.948988 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"91f948cc-e301-4b86-a1e8-a74ac8fba5ed","Type":"ContainerStarted","Data":"fc81aa11032b938f194de8a6b8264df6f85b09f507f55240c4089b53432fffe6"} Sep 10 15:44:31.183160 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:31.182993 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8e44c28-d988-4f54-8e09-363cf3cf55a2" path="/var/lib/kubelet/pods/b8e44c28-d988-4f54-8e09-363cf3cf55a2/volumes" Sep 10 15:44:31.957232 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:31.957162 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"91f948cc-e301-4b86-a1e8-a74ac8fba5ed","Type":"ContainerStarted","Data":"d657b7fefb983cc18de281d72bbad59acefe02f1a7c11729954f55f4353c5ab0"} Sep 10 15:44:31.957354 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:31.957283 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"91f948cc-e301-4b86-a1e8-a74ac8fba5ed","Type":"ContainerStarted","Data":"cc86075f010472bc58eaa9df9639c827dd47c7b34582d671259d80a4db86aad5"} Sep 10 15:44:31.957354 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:31.957330 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"91f948cc-e301-4b86-a1e8-a74ac8fba5ed","Type":"ContainerStarted","Data":"92f099d67d0b9b94b500955f521bbcbf6213edee41a840861dca780243203f85"} Sep 10 15:44:31.957458 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:31.957360 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"91f948cc-e301-4b86-a1e8-a74ac8fba5ed","Type":"ContainerStarted","Data":"eff6697a4ea8838b3f09168fffd18d4b900bef103948edf3b6eff2ad2c593ff3"} Sep 10 15:44:31.957458 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:31.957387 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"91f948cc-e301-4b86-a1e8-a74ac8fba5ed","Type":"ContainerStarted","Data":"7214b39338e908c78789a4a8fea2c4d2a67f7bbc8e79575ddc97cd283b60cdb6"} Sep 10 15:44:31.957458 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:31.957420 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"91f948cc-e301-4b86-a1e8-a74ac8fba5ed","Type":"ContainerStarted","Data":"05553ff63eb426f03571a1655ffee16543b6e3f8b627de50d11d7120840331b7"} Sep 10 15:44:31.959389 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:31.959352 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5b66c47b87-khlqk" event={"ID":"a9aa86db-f046-49a4-bba7-3a9ca2bdae16","Type":"ContainerStarted","Data":"088a28bdfac148a6cb937f5e62446721cfe2796e1a6aa84271534c7721c63582"} Sep 10 15:44:31.959512 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:31.959410 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5b66c47b87-khlqk" event={"ID":"a9aa86db-f046-49a4-bba7-3a9ca2bdae16","Type":"ContainerStarted","Data":"447141fd5bb188717ff74c7be4bcdb0b74ee4f268ad57bdcb4dfd842ab606fab"} Sep 10 15:44:31.959512 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:31.959439 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5b66c47b87-khlqk" event={"ID":"a9aa86db-f046-49a4-bba7-3a9ca2bdae16","Type":"ContainerStarted","Data":"eaec6bb1d5ce8fe8bd659f42ac429ed65d19c3a38e43651b2558d34060a61aa8"} Sep 10 15:44:31.989125 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:31.989066 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.965525135 podStartE2EDuration="2.989047964s" totalImagesPullingTime="0s" totalInitContainerRuntime="23.522829ms" isStatefulPod=false podCreationTimestamp="2026-09-10 15:44:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:44:31.987514386 +0000 UTC m=+149.341033563" watchObservedRunningTime="2026-09-10 15:44:31.989047964 +0000 UTC m=+149.342567127" Sep 10 15:44:32.011087 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:32.011047 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5b66c47b87-khlqk" podStartSLOduration=0.998616581 podStartE2EDuration="3.011034046s" totalImagesPullingTime="2.012417465s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:44:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:44:32.01018278 +0000 UTC m=+149.363701981" watchObservedRunningTime="2026-09-10 15:44:32.011034046 +0000 UTC m=+149.364553208" Sep 10 15:44:32.095696 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:32.095666 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7445646d7d-vrps2"] Sep 10 15:44:32.098774 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:32.098756 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7445646d7d-vrps2" Sep 10 15:44:32.115874 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:32.115853 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7445646d7d-vrps2"] Sep 10 15:44:32.166467 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:32.166438 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a8bd6f7-e0a8-489a-9a24-2528f79d4381-trusted-ca-bundle\") pod \"console-7445646d7d-vrps2\" (UID: \"7a8bd6f7-e0a8-489a-9a24-2528f79d4381\") " pod="openshift-console/console-7445646d7d-vrps2" Sep 10 15:44:32.166559 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:32.166472 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7a8bd6f7-e0a8-489a-9a24-2528f79d4381-oauth-serving-cert\") pod \"console-7445646d7d-vrps2\" (UID: \"7a8bd6f7-e0a8-489a-9a24-2528f79d4381\") " pod="openshift-console/console-7445646d7d-vrps2" Sep 10 15:44:32.166559 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:32.166489 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv78r\" (UniqueName: \"kubernetes.io/projected/7a8bd6f7-e0a8-489a-9a24-2528f79d4381-kube-api-access-sv78r\") pod \"console-7445646d7d-vrps2\" (UID: \"7a8bd6f7-e0a8-489a-9a24-2528f79d4381\") " pod="openshift-console/console-7445646d7d-vrps2" Sep 10 15:44:32.166559 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:32.166514 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7a8bd6f7-e0a8-489a-9a24-2528f79d4381-service-ca\") pod \"console-7445646d7d-vrps2\" (UID: \"7a8bd6f7-e0a8-489a-9a24-2528f79d4381\") " pod="openshift-console/console-7445646d7d-vrps2" Sep 10 15:44:32.166665 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:32.166560 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7a8bd6f7-e0a8-489a-9a24-2528f79d4381-console-oauth-config\") pod \"console-7445646d7d-vrps2\" (UID: \"7a8bd6f7-e0a8-489a-9a24-2528f79d4381\") " pod="openshift-console/console-7445646d7d-vrps2" Sep 10 15:44:32.166665 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:32.166583 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7a8bd6f7-e0a8-489a-9a24-2528f79d4381-console-config\") pod \"console-7445646d7d-vrps2\" (UID: \"7a8bd6f7-e0a8-489a-9a24-2528f79d4381\") " pod="openshift-console/console-7445646d7d-vrps2" Sep 10 15:44:32.166665 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:32.166607 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a8bd6f7-e0a8-489a-9a24-2528f79d4381-console-serving-cert\") pod \"console-7445646d7d-vrps2\" (UID: \"7a8bd6f7-e0a8-489a-9a24-2528f79d4381\") " pod="openshift-console/console-7445646d7d-vrps2" Sep 10 15:44:32.172641 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:32.172623 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:44:32.267528 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:32.267475 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7a8bd6f7-e0a8-489a-9a24-2528f79d4381-console-oauth-config\") pod \"console-7445646d7d-vrps2\" (UID: \"7a8bd6f7-e0a8-489a-9a24-2528f79d4381\") " pod="openshift-console/console-7445646d7d-vrps2" Sep 10 15:44:32.267528 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:32.267504 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7a8bd6f7-e0a8-489a-9a24-2528f79d4381-console-config\") pod \"console-7445646d7d-vrps2\" (UID: \"7a8bd6f7-e0a8-489a-9a24-2528f79d4381\") " pod="openshift-console/console-7445646d7d-vrps2" Sep 10 15:44:32.267528 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:32.267521 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a8bd6f7-e0a8-489a-9a24-2528f79d4381-console-serving-cert\") pod \"console-7445646d7d-vrps2\" (UID: \"7a8bd6f7-e0a8-489a-9a24-2528f79d4381\") " pod="openshift-console/console-7445646d7d-vrps2" Sep 10 15:44:32.268000 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:32.267820 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a8bd6f7-e0a8-489a-9a24-2528f79d4381-trusted-ca-bundle\") pod \"console-7445646d7d-vrps2\" (UID: \"7a8bd6f7-e0a8-489a-9a24-2528f79d4381\") " pod="openshift-console/console-7445646d7d-vrps2" Sep 10 15:44:32.268000 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:32.267860 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7a8bd6f7-e0a8-489a-9a24-2528f79d4381-oauth-serving-cert\") pod \"console-7445646d7d-vrps2\" (UID: \"7a8bd6f7-e0a8-489a-9a24-2528f79d4381\") " pod="openshift-console/console-7445646d7d-vrps2" Sep 10 15:44:32.268000 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:32.267885 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv78r\" (UniqueName: \"kubernetes.io/projected/7a8bd6f7-e0a8-489a-9a24-2528f79d4381-kube-api-access-sv78r\") pod \"console-7445646d7d-vrps2\" (UID: \"7a8bd6f7-e0a8-489a-9a24-2528f79d4381\") " pod="openshift-console/console-7445646d7d-vrps2" Sep 10 15:44:32.268000 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:32.267914 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7a8bd6f7-e0a8-489a-9a24-2528f79d4381-service-ca\") pod \"console-7445646d7d-vrps2\" (UID: \"7a8bd6f7-e0a8-489a-9a24-2528f79d4381\") " pod="openshift-console/console-7445646d7d-vrps2" Sep 10 15:44:32.268383 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:32.268324 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7a8bd6f7-e0a8-489a-9a24-2528f79d4381-console-config\") pod \"console-7445646d7d-vrps2\" (UID: \"7a8bd6f7-e0a8-489a-9a24-2528f79d4381\") " pod="openshift-console/console-7445646d7d-vrps2" Sep 10 15:44:32.268701 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:32.268679 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7a8bd6f7-e0a8-489a-9a24-2528f79d4381-service-ca\") pod \"console-7445646d7d-vrps2\" (UID: \"7a8bd6f7-e0a8-489a-9a24-2528f79d4381\") " pod="openshift-console/console-7445646d7d-vrps2" Sep 10 15:44:32.268793 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:32.268776 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7a8bd6f7-e0a8-489a-9a24-2528f79d4381-oauth-serving-cert\") pod \"console-7445646d7d-vrps2\" (UID: \"7a8bd6f7-e0a8-489a-9a24-2528f79d4381\") " pod="openshift-console/console-7445646d7d-vrps2" Sep 10 15:44:32.269299 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:32.269276 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a8bd6f7-e0a8-489a-9a24-2528f79d4381-trusted-ca-bundle\") pod \"console-7445646d7d-vrps2\" (UID: \"7a8bd6f7-e0a8-489a-9a24-2528f79d4381\") " pod="openshift-console/console-7445646d7d-vrps2" Sep 10 15:44:32.270120 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:32.270100 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7a8bd6f7-e0a8-489a-9a24-2528f79d4381-console-oauth-config\") pod \"console-7445646d7d-vrps2\" (UID: \"7a8bd6f7-e0a8-489a-9a24-2528f79d4381\") " pod="openshift-console/console-7445646d7d-vrps2" Sep 10 15:44:32.270375 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:32.270355 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a8bd6f7-e0a8-489a-9a24-2528f79d4381-console-serving-cert\") pod \"console-7445646d7d-vrps2\" (UID: \"7a8bd6f7-e0a8-489a-9a24-2528f79d4381\") " pod="openshift-console/console-7445646d7d-vrps2" Sep 10 15:44:32.275459 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:32.275439 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv78r\" (UniqueName: \"kubernetes.io/projected/7a8bd6f7-e0a8-489a-9a24-2528f79d4381-kube-api-access-sv78r\") pod \"console-7445646d7d-vrps2\" (UID: \"7a8bd6f7-e0a8-489a-9a24-2528f79d4381\") " pod="openshift-console/console-7445646d7d-vrps2" Sep 10 15:44:32.406873 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:32.406848 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7445646d7d-vrps2" Sep 10 15:44:32.687858 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:32.687830 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7445646d7d-vrps2"] Sep 10 15:44:32.690358 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:44:32.690335 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a8bd6f7_e0a8_489a_9a24_2528f79d4381.slice/crio-edb7f7fb8fc91f5b40905067fccdd83a67515cbb9b83d8cff0684c892d73c7ce WatchSource:0}: Error finding container edb7f7fb8fc91f5b40905067fccdd83a67515cbb9b83d8cff0684c892d73c7ce: Status 404 returned error can't find the container with id edb7f7fb8fc91f5b40905067fccdd83a67515cbb9b83d8cff0684c892d73c7ce Sep 10 15:44:32.964579 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:32.964496 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7445646d7d-vrps2" event={"ID":"7a8bd6f7-e0a8-489a-9a24-2528f79d4381","Type":"ContainerStarted","Data":"ac4b68c2bd91cb154f71e23d8780c1e0d52c2223e80fdfcd6fb14da28d5f9b74"} Sep 10 15:44:32.964579 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:32.964556 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7445646d7d-vrps2" event={"ID":"7a8bd6f7-e0a8-489a-9a24-2528f79d4381","Type":"ContainerStarted","Data":"edb7f7fb8fc91f5b40905067fccdd83a67515cbb9b83d8cff0684c892d73c7ce"} Sep 10 15:44:32.983984 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:32.983906 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7445646d7d-vrps2" podStartSLOduration=0.983888126 podStartE2EDuration="983.888126ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:44:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:44:32.983564993 +0000 UTC m=+150.337084178" watchObservedRunningTime="2026-09-10 15:44:32.983888126 +0000 UTC m=+150.337407289" Sep 10 15:44:35.316821 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:35.316784 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:44:42.407566 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:42.407527 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7445646d7d-vrps2" Sep 10 15:44:42.407971 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:42.407606 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7445646d7d-vrps2" Sep 10 15:44:42.412521 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:42.412496 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7445646d7d-vrps2" Sep 10 15:44:43.009182 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:43.009146 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7445646d7d-vrps2" Sep 10 15:44:43.058649 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:43.058619 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-648dccb4c6-2nc2p"] Sep 10 15:44:43.185263 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:43.185233 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:44:43.282338 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:43.282254 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:44:43.382675 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:44:43.382641 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:45:08.080435 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:08.080351 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-648dccb4c6-2nc2p" podUID="3630e074-3c8f-49f1-8c97-0dab8c8d867b" containerName="console" containerID="cri-o://3102422383c1aae668fc56b14416b61d6a24fe4d85e7441ace75398e6655371b" gracePeriod=15 Sep 10 15:45:08.316536 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:08.316512 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-648dccb4c6-2nc2p" Sep 10 15:45:08.350659 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:08.350605 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3630e074-3c8f-49f1-8c97-0dab8c8d867b-trusted-ca-bundle\") pod \"3630e074-3c8f-49f1-8c97-0dab8c8d867b\" (UID: \"3630e074-3c8f-49f1-8c97-0dab8c8d867b\") " Sep 10 15:45:08.350790 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:08.350671 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3630e074-3c8f-49f1-8c97-0dab8c8d867b-console-oauth-config\") pod \"3630e074-3c8f-49f1-8c97-0dab8c8d867b\" (UID: \"3630e074-3c8f-49f1-8c97-0dab8c8d867b\") " Sep 10 15:45:08.350833 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:08.350784 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxxkf\" (UniqueName: \"kubernetes.io/projected/3630e074-3c8f-49f1-8c97-0dab8c8d867b-kube-api-access-qxxkf\") pod \"3630e074-3c8f-49f1-8c97-0dab8c8d867b\" (UID: \"3630e074-3c8f-49f1-8c97-0dab8c8d867b\") " Sep 10 15:45:08.350867 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:08.350828 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3630e074-3c8f-49f1-8c97-0dab8c8d867b-oauth-serving-cert\") pod \"3630e074-3c8f-49f1-8c97-0dab8c8d867b\" (UID: \"3630e074-3c8f-49f1-8c97-0dab8c8d867b\") " Sep 10 15:45:08.350955 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:08.350936 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3630e074-3c8f-49f1-8c97-0dab8c8d867b-console-config\") pod \"3630e074-3c8f-49f1-8c97-0dab8c8d867b\" (UID: \"3630e074-3c8f-49f1-8c97-0dab8c8d867b\") " Sep 10 15:45:08.351017 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:08.350980 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3630e074-3c8f-49f1-8c97-0dab8c8d867b-service-ca\") pod \"3630e074-3c8f-49f1-8c97-0dab8c8d867b\" (UID: \"3630e074-3c8f-49f1-8c97-0dab8c8d867b\") " Sep 10 15:45:08.351073 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:08.351031 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3630e074-3c8f-49f1-8c97-0dab8c8d867b-console-serving-cert\") pod \"3630e074-3c8f-49f1-8c97-0dab8c8d867b\" (UID: \"3630e074-3c8f-49f1-8c97-0dab8c8d867b\") " Sep 10 15:45:08.351125 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:08.351101 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3630e074-3c8f-49f1-8c97-0dab8c8d867b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "3630e074-3c8f-49f1-8c97-0dab8c8d867b" (UID: "3630e074-3c8f-49f1-8c97-0dab8c8d867b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:45:08.351342 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:08.351161 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3630e074-3c8f-49f1-8c97-0dab8c8d867b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3630e074-3c8f-49f1-8c97-0dab8c8d867b" (UID: "3630e074-3c8f-49f1-8c97-0dab8c8d867b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:45:08.351342 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:08.351300 2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3630e074-3c8f-49f1-8c97-0dab8c8d867b-trusted-ca-bundle\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:45:08.351342 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:08.351332 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3630e074-3c8f-49f1-8c97-0dab8c8d867b-console-config" (OuterVolumeSpecName: "console-config") pod "3630e074-3c8f-49f1-8c97-0dab8c8d867b" (UID: "3630e074-3c8f-49f1-8c97-0dab8c8d867b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:45:08.351590 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:08.351570 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3630e074-3c8f-49f1-8c97-0dab8c8d867b-service-ca" (OuterVolumeSpecName: "service-ca") pod "3630e074-3c8f-49f1-8c97-0dab8c8d867b" (UID: "3630e074-3c8f-49f1-8c97-0dab8c8d867b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:45:08.353065 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:08.353038 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3630e074-3c8f-49f1-8c97-0dab8c8d867b-kube-api-access-qxxkf" (OuterVolumeSpecName: "kube-api-access-qxxkf") pod "3630e074-3c8f-49f1-8c97-0dab8c8d867b" (UID: "3630e074-3c8f-49f1-8c97-0dab8c8d867b"). InnerVolumeSpecName "kube-api-access-qxxkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:45:08.353170 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:08.353132 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3630e074-3c8f-49f1-8c97-0dab8c8d867b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3630e074-3c8f-49f1-8c97-0dab8c8d867b" (UID: "3630e074-3c8f-49f1-8c97-0dab8c8d867b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:45:08.353522 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:08.353500 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3630e074-3c8f-49f1-8c97-0dab8c8d867b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3630e074-3c8f-49f1-8c97-0dab8c8d867b" (UID: "3630e074-3c8f-49f1-8c97-0dab8c8d867b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:45:08.451929 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:08.451889 2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3630e074-3c8f-49f1-8c97-0dab8c8d867b-console-serving-cert\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:45:08.451929 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:08.451917 2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3630e074-3c8f-49f1-8c97-0dab8c8d867b-console-oauth-config\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:45:08.451929 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:08.451927 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxxkf\" (UniqueName: \"kubernetes.io/projected/3630e074-3c8f-49f1-8c97-0dab8c8d867b-kube-api-access-qxxkf\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:45:08.451929 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:08.451936 2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3630e074-3c8f-49f1-8c97-0dab8c8d867b-oauth-serving-cert\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:45:08.452179 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:08.451945 2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3630e074-3c8f-49f1-8c97-0dab8c8d867b-console-config\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:45:08.452179 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:08.451954 2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3630e074-3c8f-49f1-8c97-0dab8c8d867b-service-ca\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:45:09.107145 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:09.107105 2576 generic.go:356] "Generic (PLEG): container finished" podID="3630e074-3c8f-49f1-8c97-0dab8c8d867b" containerID="3102422383c1aae668fc56b14416b61d6a24fe4d85e7441ace75398e6655371b" exitCode=0 Sep 10 15:45:09.107576 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:09.107164 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-648dccb4c6-2nc2p" Sep 10 15:45:09.107576 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:09.107200 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-648dccb4c6-2nc2p" event={"ID":"3630e074-3c8f-49f1-8c97-0dab8c8d867b","Type":"ContainerDied","Data":"3102422383c1aae668fc56b14416b61d6a24fe4d85e7441ace75398e6655371b"} Sep 10 15:45:09.107576 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:09.107277 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-648dccb4c6-2nc2p" event={"ID":"3630e074-3c8f-49f1-8c97-0dab8c8d867b","Type":"ContainerDied","Data":"777f76761069c4b3dc6bba5f43b95a02c5c1c009b76f08512c2608d1655cbb99"} Sep 10 15:45:09.107576 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:09.107302 2576 scope.go:117] "RemoveContainer" containerID="3102422383c1aae668fc56b14416b61d6a24fe4d85e7441ace75398e6655371b" Sep 10 15:45:09.115512 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:09.115496 2576 scope.go:117] "RemoveContainer" containerID="3102422383c1aae668fc56b14416b61d6a24fe4d85e7441ace75398e6655371b" Sep 10 15:45:09.115795 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:45:09.115776 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3102422383c1aae668fc56b14416b61d6a24fe4d85e7441ace75398e6655371b\": container with ID starting with 3102422383c1aae668fc56b14416b61d6a24fe4d85e7441ace75398e6655371b not found: ID does not exist" containerID="3102422383c1aae668fc56b14416b61d6a24fe4d85e7441ace75398e6655371b" Sep 10 15:45:09.115852 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:09.115804 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3102422383c1aae668fc56b14416b61d6a24fe4d85e7441ace75398e6655371b"} err="failed to get container status \"3102422383c1aae668fc56b14416b61d6a24fe4d85e7441ace75398e6655371b\": rpc error: code = NotFound desc = could not find container \"3102422383c1aae668fc56b14416b61d6a24fe4d85e7441ace75398e6655371b\": container with ID starting with 3102422383c1aae668fc56b14416b61d6a24fe4d85e7441ace75398e6655371b not found: ID does not exist" Sep 10 15:45:09.127394 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:09.127373 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-648dccb4c6-2nc2p"] Sep 10 15:45:09.131055 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:09.131035 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-648dccb4c6-2nc2p"] Sep 10 15:45:09.167422 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:09.167394 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3630e074-3c8f-49f1-8c97-0dab8c8d867b" path="/var/lib/kubelet/pods/3630e074-3c8f-49f1-8c97-0dab8c8d867b/volumes" Sep 10 15:45:18.450154 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:18.450117 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:45:18.551433 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:18.551397 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:45:22.373090 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:22.373042 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:45:22.445042 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:22.445004 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:45:30.316727 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:30.316683 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:45:30.332819 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:30.332793 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:45:31.201189 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:31.201162 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:45:34.701636 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:34.701603 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:45:39.318802 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:39.318760 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:45:39.393169 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:39.393138 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:45:47.308243 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:47.308210 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-77585476db-g2dkb"] Sep 10 15:45:47.308634 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:47.308489 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3630e074-3c8f-49f1-8c97-0dab8c8d867b" containerName="console" Sep 10 15:45:47.308634 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:47.308500 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3630e074-3c8f-49f1-8c97-0dab8c8d867b" containerName="console" Sep 10 15:45:47.308634 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:47.308550 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3630e074-3c8f-49f1-8c97-0dab8c8d867b" containerName="console" Sep 10 15:45:47.313175 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:47.313156 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77585476db-g2dkb" Sep 10 15:45:47.336250 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:47.336224 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77585476db-g2dkb"] Sep 10 15:45:47.373829 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:47.373715 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:45:47.374799 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:47.374778 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7c7adc7e-cee3-41a0-9c55-61f882ff454a-console-oauth-config\") pod \"console-77585476db-g2dkb\" (UID: \"7c7adc7e-cee3-41a0-9c55-61f882ff454a\") " pod="openshift-console/console-77585476db-g2dkb" Sep 10 15:45:47.374912 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:47.374805 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c7adc7e-cee3-41a0-9c55-61f882ff454a-trusted-ca-bundle\") pod \"console-77585476db-g2dkb\" (UID: \"7c7adc7e-cee3-41a0-9c55-61f882ff454a\") " pod="openshift-console/console-77585476db-g2dkb" Sep 10 15:45:47.374912 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:47.374835 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c7adc7e-cee3-41a0-9c55-61f882ff454a-console-serving-cert\") pod \"console-77585476db-g2dkb\" (UID: \"7c7adc7e-cee3-41a0-9c55-61f882ff454a\") " pod="openshift-console/console-77585476db-g2dkb" Sep 10 15:45:47.374912 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:47.374883 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7c7adc7e-cee3-41a0-9c55-61f882ff454a-service-ca\") pod \"console-77585476db-g2dkb\" (UID: \"7c7adc7e-cee3-41a0-9c55-61f882ff454a\") " pod="openshift-console/console-77585476db-g2dkb" Sep 10 15:45:47.375070 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:47.374921 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v24hn\" (UniqueName: \"kubernetes.io/projected/7c7adc7e-cee3-41a0-9c55-61f882ff454a-kube-api-access-v24hn\") pod \"console-77585476db-g2dkb\" (UID: \"7c7adc7e-cee3-41a0-9c55-61f882ff454a\") " pod="openshift-console/console-77585476db-g2dkb" Sep 10 15:45:47.375070 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:47.374951 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7c7adc7e-cee3-41a0-9c55-61f882ff454a-console-config\") pod \"console-77585476db-g2dkb\" (UID: \"7c7adc7e-cee3-41a0-9c55-61f882ff454a\") " pod="openshift-console/console-77585476db-g2dkb" Sep 10 15:45:47.375070 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:47.374988 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7c7adc7e-cee3-41a0-9c55-61f882ff454a-oauth-serving-cert\") pod \"console-77585476db-g2dkb\" (UID: \"7c7adc7e-cee3-41a0-9c55-61f882ff454a\") " pod="openshift-console/console-77585476db-g2dkb" Sep 10 15:45:47.439106 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:47.439079 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:45:47.476148 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:47.476125 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7c7adc7e-cee3-41a0-9c55-61f882ff454a-console-oauth-config\") pod \"console-77585476db-g2dkb\" (UID: \"7c7adc7e-cee3-41a0-9c55-61f882ff454a\") " pod="openshift-console/console-77585476db-g2dkb" Sep 10 15:45:47.476213 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:47.476153 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c7adc7e-cee3-41a0-9c55-61f882ff454a-trusted-ca-bundle\") pod \"console-77585476db-g2dkb\" (UID: \"7c7adc7e-cee3-41a0-9c55-61f882ff454a\") " pod="openshift-console/console-77585476db-g2dkb" Sep 10 15:45:47.476213 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:47.476177 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c7adc7e-cee3-41a0-9c55-61f882ff454a-console-serving-cert\") pod \"console-77585476db-g2dkb\" (UID: \"7c7adc7e-cee3-41a0-9c55-61f882ff454a\") " pod="openshift-console/console-77585476db-g2dkb" Sep 10 15:45:47.476213 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:47.476209 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7c7adc7e-cee3-41a0-9c55-61f882ff454a-service-ca\") pod \"console-77585476db-g2dkb\" (UID: \"7c7adc7e-cee3-41a0-9c55-61f882ff454a\") " pod="openshift-console/console-77585476db-g2dkb" Sep 10 15:45:47.476421 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:47.476396 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v24hn\" (UniqueName: \"kubernetes.io/projected/7c7adc7e-cee3-41a0-9c55-61f882ff454a-kube-api-access-v24hn\") pod \"console-77585476db-g2dkb\" (UID: \"7c7adc7e-cee3-41a0-9c55-61f882ff454a\") " pod="openshift-console/console-77585476db-g2dkb" Sep 10 15:45:47.476480 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:47.476464 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7c7adc7e-cee3-41a0-9c55-61f882ff454a-console-config\") pod \"console-77585476db-g2dkb\" (UID: \"7c7adc7e-cee3-41a0-9c55-61f882ff454a\") " pod="openshift-console/console-77585476db-g2dkb" Sep 10 15:45:47.476525 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:47.476493 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7c7adc7e-cee3-41a0-9c55-61f882ff454a-oauth-serving-cert\") pod \"console-77585476db-g2dkb\" (UID: \"7c7adc7e-cee3-41a0-9c55-61f882ff454a\") " pod="openshift-console/console-77585476db-g2dkb" Sep 10 15:45:47.477010 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:47.476979 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7c7adc7e-cee3-41a0-9c55-61f882ff454a-service-ca\") pod \"console-77585476db-g2dkb\" (UID: \"7c7adc7e-cee3-41a0-9c55-61f882ff454a\") " pod="openshift-console/console-77585476db-g2dkb" Sep 10 15:45:47.477110 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:47.477028 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c7adc7e-cee3-41a0-9c55-61f882ff454a-trusted-ca-bundle\") pod \"console-77585476db-g2dkb\" (UID: \"7c7adc7e-cee3-41a0-9c55-61f882ff454a\") " pod="openshift-console/console-77585476db-g2dkb" Sep 10 15:45:47.477214 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:47.477199 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7c7adc7e-cee3-41a0-9c55-61f882ff454a-oauth-serving-cert\") pod \"console-77585476db-g2dkb\" (UID: \"7c7adc7e-cee3-41a0-9c55-61f882ff454a\") " pod="openshift-console/console-77585476db-g2dkb" Sep 10 15:45:47.477364 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:47.477341 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7c7adc7e-cee3-41a0-9c55-61f882ff454a-console-config\") pod \"console-77585476db-g2dkb\" (UID: \"7c7adc7e-cee3-41a0-9c55-61f882ff454a\") " pod="openshift-console/console-77585476db-g2dkb" Sep 10 15:45:47.478896 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:47.478868 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c7adc7e-cee3-41a0-9c55-61f882ff454a-console-serving-cert\") pod \"console-77585476db-g2dkb\" (UID: \"7c7adc7e-cee3-41a0-9c55-61f882ff454a\") " pod="openshift-console/console-77585476db-g2dkb" Sep 10 15:45:47.478972 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:47.478903 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7c7adc7e-cee3-41a0-9c55-61f882ff454a-console-oauth-config\") pod \"console-77585476db-g2dkb\" (UID: \"7c7adc7e-cee3-41a0-9c55-61f882ff454a\") " pod="openshift-console/console-77585476db-g2dkb" Sep 10 15:45:47.487511 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:47.487494 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v24hn\" (UniqueName: \"kubernetes.io/projected/7c7adc7e-cee3-41a0-9c55-61f882ff454a-kube-api-access-v24hn\") pod \"console-77585476db-g2dkb\" (UID: \"7c7adc7e-cee3-41a0-9c55-61f882ff454a\") " pod="openshift-console/console-77585476db-g2dkb" Sep 10 15:45:47.516537 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:47.516508 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:45:47.622595 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:47.622527 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77585476db-g2dkb" Sep 10 15:45:47.900918 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:47.900837 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77585476db-g2dkb"] Sep 10 15:45:47.904895 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:45:47.904865 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c7adc7e_cee3_41a0_9c55_61f882ff454a.slice/crio-87e83dbfd4cceba5f2692d0e5712d33801c8dc82808cabba514027e223730e1a WatchSource:0}: Error finding container 87e83dbfd4cceba5f2692d0e5712d33801c8dc82808cabba514027e223730e1a: Status 404 returned error can't find the container with id 87e83dbfd4cceba5f2692d0e5712d33801c8dc82808cabba514027e223730e1a Sep 10 15:45:48.109675 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:48.109640 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:45:48.180606 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:48.180524 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:45:48.247029 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:48.246997 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77585476db-g2dkb" event={"ID":"7c7adc7e-cee3-41a0-9c55-61f882ff454a","Type":"ContainerStarted","Data":"fc0cec40f665b2a5c6bbd550309cec757f57e7aebfdaba8bd33efa6094b6a5c7"} Sep 10 15:45:48.247155 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:48.247045 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77585476db-g2dkb" event={"ID":"7c7adc7e-cee3-41a0-9c55-61f882ff454a","Type":"ContainerStarted","Data":"87e83dbfd4cceba5f2692d0e5712d33801c8dc82808cabba514027e223730e1a"} Sep 10 15:45:48.275248 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:48.275205 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-77585476db-g2dkb" podStartSLOduration=1.275167857 podStartE2EDuration="1.275167857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:45:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:45:48.274044129 +0000 UTC m=+225.627563291" watchObservedRunningTime="2026-09-10 15:45:48.275167857 +0000 UTC m=+225.628687019" Sep 10 15:45:51.214031 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:51.214001 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:45:51.278572 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:51.278541 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:45:57.622901 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:57.622864 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-77585476db-g2dkb" Sep 10 15:45:57.623366 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:57.622936 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-77585476db-g2dkb" Sep 10 15:45:57.627564 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:57.627544 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-77585476db-g2dkb" Sep 10 15:45:58.288770 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:58.288711 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-77585476db-g2dkb" Sep 10 15:45:58.359020 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:58.358986 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7445646d7d-vrps2"] Sep 10 15:45:58.656129 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:58.656097 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:45:58.716865 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:45:58.716824 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:46:02.158346 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:02.158306 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:46:02.380083 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:02.380048 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:46:02.480845 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:02.480760 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:46:02.580984 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:02.580951 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:46:04.368631 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:04.368600 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:46:23.379480 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:23.379433 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7445646d7d-vrps2" podUID="7a8bd6f7-e0a8-489a-9a24-2528f79d4381" containerName="console" containerID="cri-o://ac4b68c2bd91cb154f71e23d8780c1e0d52c2223e80fdfcd6fb14da28d5f9b74" gracePeriod=15 Sep 10 15:46:23.614628 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:23.614604 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7445646d7d-vrps2" Sep 10 15:46:23.784691 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:23.784588 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sv78r\" (UniqueName: \"kubernetes.io/projected/7a8bd6f7-e0a8-489a-9a24-2528f79d4381-kube-api-access-sv78r\") pod \"7a8bd6f7-e0a8-489a-9a24-2528f79d4381\" (UID: \"7a8bd6f7-e0a8-489a-9a24-2528f79d4381\") " Sep 10 15:46:23.784691 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:23.784646 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a8bd6f7-e0a8-489a-9a24-2528f79d4381-trusted-ca-bundle\") pod \"7a8bd6f7-e0a8-489a-9a24-2528f79d4381\" (UID: \"7a8bd6f7-e0a8-489a-9a24-2528f79d4381\") " Sep 10 15:46:23.784691 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:23.784672 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a8bd6f7-e0a8-489a-9a24-2528f79d4381-console-serving-cert\") pod \"7a8bd6f7-e0a8-489a-9a24-2528f79d4381\" (UID: \"7a8bd6f7-e0a8-489a-9a24-2528f79d4381\") " Sep 10 15:46:23.784691 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:23.784699 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7a8bd6f7-e0a8-489a-9a24-2528f79d4381-console-config\") pod \"7a8bd6f7-e0a8-489a-9a24-2528f79d4381\" (UID: \"7a8bd6f7-e0a8-489a-9a24-2528f79d4381\") " Sep 10 15:46:23.785050 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:23.784721 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7a8bd6f7-e0a8-489a-9a24-2528f79d4381-service-ca\") pod \"7a8bd6f7-e0a8-489a-9a24-2528f79d4381\" (UID: \"7a8bd6f7-e0a8-489a-9a24-2528f79d4381\") " Sep 10 15:46:23.785050 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:23.784771 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7a8bd6f7-e0a8-489a-9a24-2528f79d4381-console-oauth-config\") pod \"7a8bd6f7-e0a8-489a-9a24-2528f79d4381\" (UID: \"7a8bd6f7-e0a8-489a-9a24-2528f79d4381\") " Sep 10 15:46:23.785050 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:23.784791 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7a8bd6f7-e0a8-489a-9a24-2528f79d4381-oauth-serving-cert\") pod \"7a8bd6f7-e0a8-489a-9a24-2528f79d4381\" (UID: \"7a8bd6f7-e0a8-489a-9a24-2528f79d4381\") " Sep 10 15:46:23.785233 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:23.785206 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a8bd6f7-e0a8-489a-9a24-2528f79d4381-console-config" (OuterVolumeSpecName: "console-config") pod "7a8bd6f7-e0a8-489a-9a24-2528f79d4381" (UID: "7a8bd6f7-e0a8-489a-9a24-2528f79d4381"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:46:23.785288 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:23.785245 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a8bd6f7-e0a8-489a-9a24-2528f79d4381-service-ca" (OuterVolumeSpecName: "service-ca") pod "7a8bd6f7-e0a8-489a-9a24-2528f79d4381" (UID: "7a8bd6f7-e0a8-489a-9a24-2528f79d4381"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:46:23.785288 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:23.785250 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a8bd6f7-e0a8-489a-9a24-2528f79d4381-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7a8bd6f7-e0a8-489a-9a24-2528f79d4381" (UID: "7a8bd6f7-e0a8-489a-9a24-2528f79d4381"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:46:23.785390 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:23.785287 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a8bd6f7-e0a8-489a-9a24-2528f79d4381-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7a8bd6f7-e0a8-489a-9a24-2528f79d4381" (UID: "7a8bd6f7-e0a8-489a-9a24-2528f79d4381"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:46:23.786981 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:23.786959 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a8bd6f7-e0a8-489a-9a24-2528f79d4381-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7a8bd6f7-e0a8-489a-9a24-2528f79d4381" (UID: "7a8bd6f7-e0a8-489a-9a24-2528f79d4381"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:46:23.786981 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:23.786976 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a8bd6f7-e0a8-489a-9a24-2528f79d4381-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7a8bd6f7-e0a8-489a-9a24-2528f79d4381" (UID: "7a8bd6f7-e0a8-489a-9a24-2528f79d4381"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:46:23.787111 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:23.787045 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a8bd6f7-e0a8-489a-9a24-2528f79d4381-kube-api-access-sv78r" (OuterVolumeSpecName: "kube-api-access-sv78r") pod "7a8bd6f7-e0a8-489a-9a24-2528f79d4381" (UID: "7a8bd6f7-e0a8-489a-9a24-2528f79d4381"). InnerVolumeSpecName "kube-api-access-sv78r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:46:23.886086 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:23.886050 2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7a8bd6f7-e0a8-489a-9a24-2528f79d4381-console-config\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:46:23.886086 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:23.886077 2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7a8bd6f7-e0a8-489a-9a24-2528f79d4381-service-ca\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:46:23.886086 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:23.886086 2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7a8bd6f7-e0a8-489a-9a24-2528f79d4381-console-oauth-config\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:46:23.886086 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:23.886097 2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7a8bd6f7-e0a8-489a-9a24-2528f79d4381-oauth-serving-cert\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:46:23.886337 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:23.886105 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv78r\" (UniqueName: \"kubernetes.io/projected/7a8bd6f7-e0a8-489a-9a24-2528f79d4381-kube-api-access-sv78r\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:46:23.886337 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:23.886113 2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a8bd6f7-e0a8-489a-9a24-2528f79d4381-trusted-ca-bundle\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:46:23.886337 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:23.886121 2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a8bd6f7-e0a8-489a-9a24-2528f79d4381-console-serving-cert\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:46:24.376863 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:24.376782 2576 generic.go:356] "Generic (PLEG): container finished" podID="7a8bd6f7-e0a8-489a-9a24-2528f79d4381" containerID="ac4b68c2bd91cb154f71e23d8780c1e0d52c2223e80fdfcd6fb14da28d5f9b74" exitCode=0 Sep 10 15:46:24.376863 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:24.376835 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7445646d7d-vrps2" Sep 10 15:46:24.377070 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:24.376864 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7445646d7d-vrps2" event={"ID":"7a8bd6f7-e0a8-489a-9a24-2528f79d4381","Type":"ContainerDied","Data":"ac4b68c2bd91cb154f71e23d8780c1e0d52c2223e80fdfcd6fb14da28d5f9b74"} Sep 10 15:46:24.377070 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:24.376915 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7445646d7d-vrps2" event={"ID":"7a8bd6f7-e0a8-489a-9a24-2528f79d4381","Type":"ContainerDied","Data":"edb7f7fb8fc91f5b40905067fccdd83a67515cbb9b83d8cff0684c892d73c7ce"} Sep 10 15:46:24.377070 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:24.376935 2576 scope.go:117] "RemoveContainer" containerID="ac4b68c2bd91cb154f71e23d8780c1e0d52c2223e80fdfcd6fb14da28d5f9b74" Sep 10 15:46:24.391186 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:24.391043 2576 scope.go:117] "RemoveContainer" containerID="ac4b68c2bd91cb154f71e23d8780c1e0d52c2223e80fdfcd6fb14da28d5f9b74" Sep 10 15:46:24.391417 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:46:24.391371 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac4b68c2bd91cb154f71e23d8780c1e0d52c2223e80fdfcd6fb14da28d5f9b74\": container with ID starting with ac4b68c2bd91cb154f71e23d8780c1e0d52c2223e80fdfcd6fb14da28d5f9b74 not found: ID does not exist" containerID="ac4b68c2bd91cb154f71e23d8780c1e0d52c2223e80fdfcd6fb14da28d5f9b74" Sep 10 15:46:24.391417 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:24.391396 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac4b68c2bd91cb154f71e23d8780c1e0d52c2223e80fdfcd6fb14da28d5f9b74"} err="failed to get container status \"ac4b68c2bd91cb154f71e23d8780c1e0d52c2223e80fdfcd6fb14da28d5f9b74\": rpc error: code = NotFound desc = could not find container \"ac4b68c2bd91cb154f71e23d8780c1e0d52c2223e80fdfcd6fb14da28d5f9b74\": container with ID starting with ac4b68c2bd91cb154f71e23d8780c1e0d52c2223e80fdfcd6fb14da28d5f9b74 not found: ID does not exist" Sep 10 15:46:24.403103 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:24.403071 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7445646d7d-vrps2"] Sep 10 15:46:24.406576 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:24.406551 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7445646d7d-vrps2"] Sep 10 15:46:24.636839 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:24.636754 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:46:24.742954 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:24.742923 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:46:25.167976 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:25.167939 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a8bd6f7-e0a8-489a-9a24-2528f79d4381" path="/var/lib/kubelet/pods/7a8bd6f7-e0a8-489a-9a24-2528f79d4381/volumes" Sep 10 15:46:46.750360 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:46.750320 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:46:48.253243 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:48.253200 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:46:50.563898 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:50.563870 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:46:50.676339 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:50.676304 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:46:52.532897 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:52.532861 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:46:52.721383 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:52.721347 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:46:52.770572 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:52.770536 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-x4rlm"] Sep 10 15:46:52.770882 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:52.770869 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a8bd6f7-e0a8-489a-9a24-2528f79d4381" containerName="console" Sep 10 15:46:52.770930 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:52.770884 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a8bd6f7-e0a8-489a-9a24-2528f79d4381" containerName="console" Sep 10 15:46:52.770964 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:52.770958 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a8bd6f7-e0a8-489a-9a24-2528f79d4381" containerName="console" Sep 10 15:46:52.775250 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:52.775236 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-x4rlm" Sep 10 15:46:52.777833 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:52.777812 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-5vt29\"" Sep 10 15:46:52.778252 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:52.778239 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 10 15:46:52.778491 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:52.778477 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 10 15:46:52.790902 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:52.790880 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-x4rlm"] Sep 10 15:46:52.798783 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:52.798760 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cjj5\" (UniqueName: \"kubernetes.io/projected/2c664453-099d-430d-803b-a0a811659818-kube-api-access-6cjj5\") pod \"cert-manager-webhook-6486ffd997-x4rlm\" (UID: \"2c664453-099d-430d-803b-a0a811659818\") " pod="cert-manager/cert-manager-webhook-6486ffd997-x4rlm" Sep 10 15:46:52.798866 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:52.798807 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2c664453-099d-430d-803b-a0a811659818-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-x4rlm\" (UID: \"2c664453-099d-430d-803b-a0a811659818\") " pod="cert-manager/cert-manager-webhook-6486ffd997-x4rlm" Sep 10 15:46:52.899330 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:52.899302 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cjj5\" (UniqueName: \"kubernetes.io/projected/2c664453-099d-430d-803b-a0a811659818-kube-api-access-6cjj5\") pod \"cert-manager-webhook-6486ffd997-x4rlm\" (UID: \"2c664453-099d-430d-803b-a0a811659818\") " pod="cert-manager/cert-manager-webhook-6486ffd997-x4rlm" Sep 10 15:46:52.899430 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:52.899339 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2c664453-099d-430d-803b-a0a811659818-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-x4rlm\" (UID: \"2c664453-099d-430d-803b-a0a811659818\") " pod="cert-manager/cert-manager-webhook-6486ffd997-x4rlm" Sep 10 15:46:52.918024 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:52.918002 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2c664453-099d-430d-803b-a0a811659818-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-x4rlm\" (UID: \"2c664453-099d-430d-803b-a0a811659818\") " pod="cert-manager/cert-manager-webhook-6486ffd997-x4rlm" Sep 10 15:46:52.926175 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:52.926156 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cjj5\" (UniqueName: \"kubernetes.io/projected/2c664453-099d-430d-803b-a0a811659818-kube-api-access-6cjj5\") pod \"cert-manager-webhook-6486ffd997-x4rlm\" (UID: \"2c664453-099d-430d-803b-a0a811659818\") " pod="cert-manager/cert-manager-webhook-6486ffd997-x4rlm" Sep 10 15:46:53.094466 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:53.094405 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-x4rlm" Sep 10 15:46:53.389276 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:53.389209 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-x4rlm"] Sep 10 15:46:53.392405 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:46:53.392378 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c664453_099d_430d_803b_a0a811659818.slice/crio-a4e502c7fc4b2edeb5ba69e82c375c97e99eb77fa88b19dc3aae3ad0a3d80009 WatchSource:0}: Error finding container a4e502c7fc4b2edeb5ba69e82c375c97e99eb77fa88b19dc3aae3ad0a3d80009: Status 404 returned error can't find the container with id a4e502c7fc4b2edeb5ba69e82c375c97e99eb77fa88b19dc3aae3ad0a3d80009 Sep 10 15:46:53.485117 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:53.485085 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-x4rlm" event={"ID":"2c664453-099d-430d-803b-a0a811659818","Type":"ContainerStarted","Data":"a4e502c7fc4b2edeb5ba69e82c375c97e99eb77fa88b19dc3aae3ad0a3d80009"} Sep 10 15:46:54.482039 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:54.481941 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:46:54.885860 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:54.885801 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:46:57.363293 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:57.363255 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:46:57.466442 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:57.466400 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:46:57.502211 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:57.502179 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-x4rlm" event={"ID":"2c664453-099d-430d-803b-a0a811659818","Type":"ContainerStarted","Data":"a5260d24de7d5972d3ca1ef812b61e70d6e2d0dc16dffadb640bce83e562b8eb"} Sep 10 15:46:57.502398 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:57.502298 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-x4rlm" Sep 10 15:46:57.519177 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:46:57.519118 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-x4rlm" podStartSLOduration=2.44947767 podStartE2EDuration="5.519100172s" totalImagesPullingTime="3.069622502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:46:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:46:57.518974769 +0000 UTC m=+294.872493933" watchObservedRunningTime="2026-09-10 15:46:57.519100172 +0000 UTC m=+294.872619331" Sep 10 15:47:02.281248 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:02.281209 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:03.081749 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:03.081704 2576 kubelet.go:1648] "Image garbage collection succeeded" Sep 10 15:47:03.508627 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:03.508555 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-x4rlm" Sep 10 15:47:03.570789 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:03.570760 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:03.672441 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:03.672406 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:05.652973 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:05.652937 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:05.784783 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:05.784727 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:06.815882 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:06.815853 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pjmnk"] Sep 10 15:47:06.818806 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:06.818789 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pjmnk" Sep 10 15:47:06.821136 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:06.821114 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 10 15:47:06.821255 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:06.821135 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 10 15:47:06.821887 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:06.821870 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-7vt7g\"" Sep 10 15:47:06.829279 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:06.829259 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pjmnk"] Sep 10 15:47:06.912049 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:06.912015 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcm6h\" (UniqueName: \"kubernetes.io/projected/fd5e7781-339a-418a-aa14-6b1ae3ff816a-kube-api-access-hcm6h\") pod \"openshift-lws-operator-fd8ccff4c-pjmnk\" (UID: \"fd5e7781-339a-418a-aa14-6b1ae3ff816a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pjmnk" Sep 10 15:47:06.912186 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:06.912064 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd5e7781-339a-418a-aa14-6b1ae3ff816a-tmp\") pod \"openshift-lws-operator-fd8ccff4c-pjmnk\" (UID: \"fd5e7781-339a-418a-aa14-6b1ae3ff816a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pjmnk" Sep 10 15:47:07.013381 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:07.013352 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcm6h\" (UniqueName: \"kubernetes.io/projected/fd5e7781-339a-418a-aa14-6b1ae3ff816a-kube-api-access-hcm6h\") pod \"openshift-lws-operator-fd8ccff4c-pjmnk\" (UID: \"fd5e7781-339a-418a-aa14-6b1ae3ff816a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pjmnk" Sep 10 15:47:07.013496 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:07.013390 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd5e7781-339a-418a-aa14-6b1ae3ff816a-tmp\") pod \"openshift-lws-operator-fd8ccff4c-pjmnk\" (UID: \"fd5e7781-339a-418a-aa14-6b1ae3ff816a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pjmnk" Sep 10 15:47:07.013711 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:07.013695 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd5e7781-339a-418a-aa14-6b1ae3ff816a-tmp\") pod \"openshift-lws-operator-fd8ccff4c-pjmnk\" (UID: \"fd5e7781-339a-418a-aa14-6b1ae3ff816a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pjmnk" Sep 10 15:47:07.021305 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:07.021282 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcm6h\" (UniqueName: \"kubernetes.io/projected/fd5e7781-339a-418a-aa14-6b1ae3ff816a-kube-api-access-hcm6h\") pod \"openshift-lws-operator-fd8ccff4c-pjmnk\" (UID: \"fd5e7781-339a-418a-aa14-6b1ae3ff816a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pjmnk" Sep 10 15:47:07.127205 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:07.127149 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pjmnk" Sep 10 15:47:07.394719 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:07.394648 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pjmnk"] Sep 10 15:47:07.398544 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:47:07.398502 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd5e7781_339a_418a_aa14_6b1ae3ff816a.slice/crio-c8c3c31604265f2fe439bc390e72a9f1e0b776d57f7970de52a400fdd4dafebc WatchSource:0}: Error finding container c8c3c31604265f2fe439bc390e72a9f1e0b776d57f7970de52a400fdd4dafebc: Status 404 returned error can't find the container with id c8c3c31604265f2fe439bc390e72a9f1e0b776d57f7970de52a400fdd4dafebc Sep 10 15:47:07.400563 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:07.400546 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 15:47:07.542289 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:07.542256 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pjmnk" event={"ID":"fd5e7781-339a-418a-aa14-6b1ae3ff816a","Type":"ContainerStarted","Data":"c8c3c31604265f2fe439bc390e72a9f1e0b776d57f7970de52a400fdd4dafebc"} Sep 10 15:47:10.556774 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:10.556710 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pjmnk" event={"ID":"fd5e7781-339a-418a-aa14-6b1ae3ff816a","Type":"ContainerStarted","Data":"d6830ecacff770fce8ad2a1b7e7a5e0a223e5dd2db15a56533989e508a07b011"} Sep 10 15:47:10.574333 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:10.574280 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pjmnk" podStartSLOduration=2.263856708 podStartE2EDuration="4.574265212s" totalImagesPullingTime="2.310408504s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:47:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:47:10.572444964 +0000 UTC m=+307.925964126" watchObservedRunningTime="2026-09-10 15:47:10.574265212 +0000 UTC m=+307.927784373" Sep 10 15:47:11.289530 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:11.289496 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:11.388882 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:11.388845 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:13.338910 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:13.338874 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:28.839499 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:28.839460 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:28.931888 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:28.931857 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:29.409192 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:29.409160 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:29.684915 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:29.684840 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:29.823585 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:29.823536 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:29.918030 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:29.918007 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-97f7948ff-cr5vg"] Sep 10 15:47:29.924960 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:29.924945 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-97f7948ff-cr5vg" Sep 10 15:47:29.927541 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:29.927511 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 10 15:47:29.928854 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:29.928829 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-wwh6s\"" Sep 10 15:47:29.929102 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:29.929079 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 10 15:47:29.929163 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:29.929140 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 10 15:47:29.929388 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:29.929370 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 10 15:47:29.950763 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:29.950718 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-97f7948ff-cr5vg"] Sep 10 15:47:30.100671 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:30.100647 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/899506ad-9415-4a67-b953-3d8134d64297-manifests\") pod \"opendatahub-operator-controller-manager-97f7948ff-cr5vg\" (UID: \"899506ad-9415-4a67-b953-3d8134d64297\") " pod="opendatahub/opendatahub-operator-controller-manager-97f7948ff-cr5vg" Sep 10 15:47:30.100815 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:30.100678 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbrkk\" (UniqueName: \"kubernetes.io/projected/899506ad-9415-4a67-b953-3d8134d64297-kube-api-access-gbrkk\") pod \"opendatahub-operator-controller-manager-97f7948ff-cr5vg\" (UID: \"899506ad-9415-4a67-b953-3d8134d64297\") " pod="opendatahub/opendatahub-operator-controller-manager-97f7948ff-cr5vg" Sep 10 15:47:30.100815 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:30.100706 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/899506ad-9415-4a67-b953-3d8134d64297-tmp\") pod \"opendatahub-operator-controller-manager-97f7948ff-cr5vg\" (UID: \"899506ad-9415-4a67-b953-3d8134d64297\") " pod="opendatahub/opendatahub-operator-controller-manager-97f7948ff-cr5vg" Sep 10 15:47:30.100894 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:30.100810 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/899506ad-9415-4a67-b953-3d8134d64297-apiservice-cert\") pod \"opendatahub-operator-controller-manager-97f7948ff-cr5vg\" (UID: \"899506ad-9415-4a67-b953-3d8134d64297\") " pod="opendatahub/opendatahub-operator-controller-manager-97f7948ff-cr5vg" Sep 10 15:47:30.100894 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:30.100865 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/899506ad-9415-4a67-b953-3d8134d64297-webhook-cert\") pod \"opendatahub-operator-controller-manager-97f7948ff-cr5vg\" (UID: \"899506ad-9415-4a67-b953-3d8134d64297\") " pod="opendatahub/opendatahub-operator-controller-manager-97f7948ff-cr5vg" Sep 10 15:47:30.156056 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:30.156031 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:30.201210 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:30.201155 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/899506ad-9415-4a67-b953-3d8134d64297-apiservice-cert\") pod \"opendatahub-operator-controller-manager-97f7948ff-cr5vg\" (UID: \"899506ad-9415-4a67-b953-3d8134d64297\") " pod="opendatahub/opendatahub-operator-controller-manager-97f7948ff-cr5vg" Sep 10 15:47:30.201210 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:30.201197 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/899506ad-9415-4a67-b953-3d8134d64297-webhook-cert\") pod \"opendatahub-operator-controller-manager-97f7948ff-cr5vg\" (UID: \"899506ad-9415-4a67-b953-3d8134d64297\") " pod="opendatahub/opendatahub-operator-controller-manager-97f7948ff-cr5vg" Sep 10 15:47:30.201338 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:30.201238 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/899506ad-9415-4a67-b953-3d8134d64297-manifests\") pod \"opendatahub-operator-controller-manager-97f7948ff-cr5vg\" (UID: \"899506ad-9415-4a67-b953-3d8134d64297\") " pod="opendatahub/opendatahub-operator-controller-manager-97f7948ff-cr5vg" Sep 10 15:47:30.201338 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:30.201261 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbrkk\" (UniqueName: \"kubernetes.io/projected/899506ad-9415-4a67-b953-3d8134d64297-kube-api-access-gbrkk\") pod \"opendatahub-operator-controller-manager-97f7948ff-cr5vg\" (UID: \"899506ad-9415-4a67-b953-3d8134d64297\") " pod="opendatahub/opendatahub-operator-controller-manager-97f7948ff-cr5vg" Sep 10 15:47:30.201338 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:30.201289 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/899506ad-9415-4a67-b953-3d8134d64297-tmp\") pod \"opendatahub-operator-controller-manager-97f7948ff-cr5vg\" (UID: \"899506ad-9415-4a67-b953-3d8134d64297\") " pod="opendatahub/opendatahub-operator-controller-manager-97f7948ff-cr5vg" Sep 10 15:47:30.201686 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:30.201664 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/899506ad-9415-4a67-b953-3d8134d64297-manifests\") pod \"opendatahub-operator-controller-manager-97f7948ff-cr5vg\" (UID: \"899506ad-9415-4a67-b953-3d8134d64297\") " pod="opendatahub/opendatahub-operator-controller-manager-97f7948ff-cr5vg" Sep 10 15:47:30.203695 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:30.203675 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/899506ad-9415-4a67-b953-3d8134d64297-tmp\") pod \"opendatahub-operator-controller-manager-97f7948ff-cr5vg\" (UID: \"899506ad-9415-4a67-b953-3d8134d64297\") " pod="opendatahub/opendatahub-operator-controller-manager-97f7948ff-cr5vg" Sep 10 15:47:30.203881 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:30.203864 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/899506ad-9415-4a67-b953-3d8134d64297-webhook-cert\") pod \"opendatahub-operator-controller-manager-97f7948ff-cr5vg\" (UID: \"899506ad-9415-4a67-b953-3d8134d64297\") " pod="opendatahub/opendatahub-operator-controller-manager-97f7948ff-cr5vg" Sep 10 15:47:30.203881 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:30.203872 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/899506ad-9415-4a67-b953-3d8134d64297-apiservice-cert\") pod \"opendatahub-operator-controller-manager-97f7948ff-cr5vg\" (UID: \"899506ad-9415-4a67-b953-3d8134d64297\") " pod="opendatahub/opendatahub-operator-controller-manager-97f7948ff-cr5vg" Sep 10 15:47:30.211794 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:30.211774 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbrkk\" (UniqueName: \"kubernetes.io/projected/899506ad-9415-4a67-b953-3d8134d64297-kube-api-access-gbrkk\") pod \"opendatahub-operator-controller-manager-97f7948ff-cr5vg\" (UID: \"899506ad-9415-4a67-b953-3d8134d64297\") " pod="opendatahub/opendatahub-operator-controller-manager-97f7948ff-cr5vg" Sep 10 15:47:30.238796 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:30.238772 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-97f7948ff-cr5vg" Sep 10 15:47:30.256254 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:30.256217 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:30.356635 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:30.356601 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:30.432113 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:30.432088 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:30.516780 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:30.516648 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-97f7948ff-cr5vg"] Sep 10 15:47:30.517885 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:30.517852 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:30.519185 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:47:30.519162 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod899506ad_9415_4a67_b953_3d8134d64297.slice/crio-5bf0cb10028ecfc8be94c7385095e617eb111548babeca75abdf1e4378de00cc WatchSource:0}: Error finding container 5bf0cb10028ecfc8be94c7385095e617eb111548babeca75abdf1e4378de00cc: Status 404 returned error can't find the container with id 5bf0cb10028ecfc8be94c7385095e617eb111548babeca75abdf1e4378de00cc Sep 10 15:47:30.612331 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:30.612308 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:30.631851 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:30.631829 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-97f7948ff-cr5vg" event={"ID":"899506ad-9415-4a67-b953-3d8134d64297","Type":"ContainerStarted","Data":"5bf0cb10028ecfc8be94c7385095e617eb111548babeca75abdf1e4378de00cc"} Sep 10 15:47:30.713774 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:30.713749 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:31.658971 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:31.658927 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:31.746265 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:31.746177 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:34.650008 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:34.649972 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-97f7948ff-cr5vg" event={"ID":"899506ad-9415-4a67-b953-3d8134d64297","Type":"ContainerStarted","Data":"d14bad7add54b5652232882b7a87bc4b461ef784321ba711097ade9f744e97c3"} Sep 10 15:47:35.655277 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:35.655243 2576 generic.go:356] "Generic (PLEG): container finished" podID="899506ad-9415-4a67-b953-3d8134d64297" containerID="d14bad7add54b5652232882b7a87bc4b461ef784321ba711097ade9f744e97c3" exitCode=0 Sep 10 15:47:35.655618 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:35.655302 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-97f7948ff-cr5vg" event={"ID":"899506ad-9415-4a67-b953-3d8134d64297","Type":"ContainerDied","Data":"d14bad7add54b5652232882b7a87bc4b461ef784321ba711097ade9f744e97c3"} Sep 10 15:47:36.660888 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:36.660853 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-97f7948ff-cr5vg" event={"ID":"899506ad-9415-4a67-b953-3d8134d64297","Type":"ContainerStarted","Data":"7c4b567e957c607b8d6dd16cfd105ec8ab78aae476c3cab16d6e01ebc4d53176"} Sep 10 15:47:36.661327 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:36.660972 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-97f7948ff-cr5vg" Sep 10 15:47:36.686704 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:36.686655 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-97f7948ff-cr5vg" podStartSLOduration=3.073429455 podStartE2EDuration="7.686642695s" totalImagesPullingTime="3.570193153s" totalInitContainerRuntime="1.043020087s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:47:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:47:36.685918173 +0000 UTC m=+334.039437352" watchObservedRunningTime="2026-09-10 15:47:36.686642695 +0000 UTC m=+334.040161855" Sep 10 15:47:47.668027 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:47.667995 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-97f7948ff-cr5vg" Sep 10 15:47:50.154216 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:50.154183 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:50.301335 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:50.301298 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:50.610655 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:50.610627 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:50.730893 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:50.730859 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-644bb569f8-dslcn"] Sep 10 15:47:50.740753 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:50.740707 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-644bb569f8-dslcn" Sep 10 15:47:50.744603 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:50.744585 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 10 15:47:50.750613 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:50.750595 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-7pq96\"" Sep 10 15:47:50.750692 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:50.750678 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 10 15:47:50.750764 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:50.750719 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 10 15:47:50.753864 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:50.753848 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 10 15:47:50.758560 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:50.758545 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xx75\" (UniqueName: \"kubernetes.io/projected/0b5650cc-b071-4c0a-a616-16012fb0a0a0-kube-api-access-6xx75\") pod \"kube-auth-proxy-644bb569f8-dslcn\" (UID: \"0b5650cc-b071-4c0a-a616-16012fb0a0a0\") " pod="openshift-ingress/kube-auth-proxy-644bb569f8-dslcn" Sep 10 15:47:50.758612 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:50.758582 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b5650cc-b071-4c0a-a616-16012fb0a0a0-tmp\") pod \"kube-auth-proxy-644bb569f8-dslcn\" (UID: \"0b5650cc-b071-4c0a-a616-16012fb0a0a0\") " pod="openshift-ingress/kube-auth-proxy-644bb569f8-dslcn" Sep 10 15:47:50.758645 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:50.758616 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b5650cc-b071-4c0a-a616-16012fb0a0a0-tls-certs\") pod \"kube-auth-proxy-644bb569f8-dslcn\" (UID: \"0b5650cc-b071-4c0a-a616-16012fb0a0a0\") " pod="openshift-ingress/kube-auth-proxy-644bb569f8-dslcn" Sep 10 15:47:50.769122 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:50.769102 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-644bb569f8-dslcn"] Sep 10 15:47:50.859632 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:50.859609 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b5650cc-b071-4c0a-a616-16012fb0a0a0-tls-certs\") pod \"kube-auth-proxy-644bb569f8-dslcn\" (UID: \"0b5650cc-b071-4c0a-a616-16012fb0a0a0\") " pod="openshift-ingress/kube-auth-proxy-644bb569f8-dslcn" Sep 10 15:47:50.859717 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:50.859666 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xx75\" (UniqueName: \"kubernetes.io/projected/0b5650cc-b071-4c0a-a616-16012fb0a0a0-kube-api-access-6xx75\") pod \"kube-auth-proxy-644bb569f8-dslcn\" (UID: \"0b5650cc-b071-4c0a-a616-16012fb0a0a0\") " pod="openshift-ingress/kube-auth-proxy-644bb569f8-dslcn" Sep 10 15:47:50.859717 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:50.859698 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b5650cc-b071-4c0a-a616-16012fb0a0a0-tmp\") pod \"kube-auth-proxy-644bb569f8-dslcn\" (UID: \"0b5650cc-b071-4c0a-a616-16012fb0a0a0\") " pod="openshift-ingress/kube-auth-proxy-644bb569f8-dslcn" Sep 10 15:47:50.859824 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:47:50.859775 2576 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Sep 10 15:47:50.859860 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:47:50.859837 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b5650cc-b071-4c0a-a616-16012fb0a0a0-tls-certs podName:0b5650cc-b071-4c0a-a616-16012fb0a0a0 nodeName:}" failed. No retries permitted until 2026-09-10 15:47:51.35981884 +0000 UTC m=+348.713337979 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/0b5650cc-b071-4c0a-a616-16012fb0a0a0-tls-certs") pod "kube-auth-proxy-644bb569f8-dslcn" (UID: "0b5650cc-b071-4c0a-a616-16012fb0a0a0") : secret "kube-auth-proxy-tls" not found Sep 10 15:47:50.862011 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:50.861975 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b5650cc-b071-4c0a-a616-16012fb0a0a0-tmp\") pod \"kube-auth-proxy-644bb569f8-dslcn\" (UID: \"0b5650cc-b071-4c0a-a616-16012fb0a0a0\") " pod="openshift-ingress/kube-auth-proxy-644bb569f8-dslcn" Sep 10 15:47:50.878744 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:50.878706 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xx75\" (UniqueName: \"kubernetes.io/projected/0b5650cc-b071-4c0a-a616-16012fb0a0a0-kube-api-access-6xx75\") pod \"kube-auth-proxy-644bb569f8-dslcn\" (UID: \"0b5650cc-b071-4c0a-a616-16012fb0a0a0\") " pod="openshift-ingress/kube-auth-proxy-644bb569f8-dslcn" Sep 10 15:47:51.003133 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:51.003101 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:51.364689 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:51.364653 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b5650cc-b071-4c0a-a616-16012fb0a0a0-tls-certs\") pod \"kube-auth-proxy-644bb569f8-dslcn\" (UID: \"0b5650cc-b071-4c0a-a616-16012fb0a0a0\") " pod="openshift-ingress/kube-auth-proxy-644bb569f8-dslcn" Sep 10 15:47:51.366980 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:51.366958 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b5650cc-b071-4c0a-a616-16012fb0a0a0-tls-certs\") pod \"kube-auth-proxy-644bb569f8-dslcn\" (UID: \"0b5650cc-b071-4c0a-a616-16012fb0a0a0\") " pod="openshift-ingress/kube-auth-proxy-644bb569f8-dslcn" Sep 10 15:47:51.650357 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:51.650289 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-644bb569f8-dslcn" Sep 10 15:47:51.924399 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:51.924375 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-644bb569f8-dslcn"] Sep 10 15:47:51.926953 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:47:51.926923 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b5650cc_b071_4c0a_a616_16012fb0a0a0.slice/crio-18b926785f6c3b459c15db58e1c48ac3161a7b35fa1ab11a3614faf665a5c1c4 WatchSource:0}: Error finding container 18b926785f6c3b459c15db58e1c48ac3161a7b35fa1ab11a3614faf665a5c1c4: Status 404 returned error can't find the container with id 18b926785f6c3b459c15db58e1c48ac3161a7b35fa1ab11a3614faf665a5c1c4 Sep 10 15:47:52.329008 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:52.328978 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:52.415374 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:52.415342 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:52.742274 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:52.742182 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-644bb569f8-dslcn" event={"ID":"0b5650cc-b071-4c0a-a616-16012fb0a0a0","Type":"ContainerStarted","Data":"18b926785f6c3b459c15db58e1c48ac3161a7b35fa1ab11a3614faf665a5c1c4"} Sep 10 15:47:53.443205 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:53.443168 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:54.752712 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:54.752680 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-644bb569f8-dslcn" event={"ID":"0b5650cc-b071-4c0a-a616-16012fb0a0a0","Type":"ContainerStarted","Data":"d2bf8ede8c0ca0685c639ee9cc11bf011bed6ed88c9d1e784ba1f1b5575dd46c"} Sep 10 15:47:54.767678 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:54.767628 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-644bb569f8-dslcn" podStartSLOduration=2.163934063 podStartE2EDuration="4.767614055s" totalImagesPullingTime="2.603679992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:47:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:47:54.767264608 +0000 UTC m=+352.120783787" watchObservedRunningTime="2026-09-10 15:47:54.767614055 +0000 UTC m=+352.121133216" Sep 10 15:47:56.880093 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:56.880059 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:56.976174 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:56.976152 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:57.122876 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.122848 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc964sf"] Sep 10 15:47:57.126006 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.125986 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc964sf" Sep 10 15:47:57.132325 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.132307 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 10 15:47:57.132403 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.132387 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-sw7wg\"" Sep 10 15:47:57.149480 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.149456 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc964sf"] Sep 10 15:47:57.173280 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.173252 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:57.208477 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.208452 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc964sf\" (UID: \"ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc964sf" Sep 10 15:47:57.208583 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.208491 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv2ww\" (UniqueName: \"kubernetes.io/projected/ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076-kube-api-access-tv2ww\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc964sf\" (UID: \"ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc964sf" Sep 10 15:47:57.208583 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.208523 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc964sf\" (UID: \"ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc964sf" Sep 10 15:47:57.208676 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.208582 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc964sf\" (UID: \"ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc964sf" Sep 10 15:47:57.208676 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.208623 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc964sf\" (UID: \"ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc964sf" Sep 10 15:47:57.208676 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.208639 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc964sf\" (UID: \"ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc964sf" Sep 10 15:47:57.208676 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.208662 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc964sf\" (UID: \"ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc964sf" Sep 10 15:47:57.208853 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.208691 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc964sf\" (UID: \"ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc964sf" Sep 10 15:47:57.208853 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.208717 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc964sf\" (UID: \"ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc964sf" Sep 10 15:47:57.211457 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.211437 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz"] Sep 10 15:47:57.214544 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.214530 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz" Sep 10 15:47:57.235094 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.235070 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz"] Sep 10 15:47:57.280254 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.280215 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:57.310084 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.310056 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df543b0d-1120-499b-8a4a-ea71ca5e539d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz\" (UID: \"df543b0d-1120-499b-8a4a-ea71ca5e539d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz" Sep 10 15:47:57.310175 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.310091 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc964sf\" (UID: \"ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc964sf" Sep 10 15:47:57.310175 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.310110 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df543b0d-1120-499b-8a4a-ea71ca5e539d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz\" (UID: \"df543b0d-1120-499b-8a4a-ea71ca5e539d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz" Sep 10 15:47:57.310175 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.310133 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc964sf\" (UID: \"ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc964sf" Sep 10 15:47:57.310175 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.310163 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df543b0d-1120-499b-8a4a-ea71ca5e539d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz\" (UID: \"df543b0d-1120-499b-8a4a-ea71ca5e539d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz" Sep 10 15:47:57.310338 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.310283 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df543b0d-1120-499b-8a4a-ea71ca5e539d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz\" (UID: \"df543b0d-1120-499b-8a4a-ea71ca5e539d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz" Sep 10 15:47:57.310338 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.310326 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv2ww\" (UniqueName: \"kubernetes.io/projected/ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076-kube-api-access-tv2ww\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc964sf\" (UID: \"ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc964sf" Sep 10 15:47:57.310423 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.310357 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljftf\" (UniqueName: \"kubernetes.io/projected/df543b0d-1120-499b-8a4a-ea71ca5e539d-kube-api-access-ljftf\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz\" (UID: \"df543b0d-1120-499b-8a4a-ea71ca5e539d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz" Sep 10 15:47:57.310423 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.310388 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc964sf\" (UID: \"ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc964sf" Sep 10 15:47:57.310423 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.310414 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df543b0d-1120-499b-8a4a-ea71ca5e539d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz\" (UID: \"df543b0d-1120-499b-8a4a-ea71ca5e539d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz" Sep 10 15:47:57.310562 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.310436 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc964sf\" (UID: \"ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc964sf" Sep 10 15:47:57.310562 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.310437 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df543b0d-1120-499b-8a4a-ea71ca5e539d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz\" (UID: \"df543b0d-1120-499b-8a4a-ea71ca5e539d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz" Sep 10 15:47:57.310562 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.310503 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc964sf\" (UID: \"ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc964sf" Sep 10 15:47:57.310562 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.310536 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc964sf\" (UID: \"ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc964sf" Sep 10 15:47:57.310562 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.310553 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc964sf\" (UID: \"ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc964sf" Sep 10 15:47:57.310844 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.310582 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc964sf\" (UID: \"ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc964sf" Sep 10 15:47:57.310844 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.310609 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc964sf\" (UID: \"ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc964sf" Sep 10 15:47:57.310844 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.310636 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df543b0d-1120-499b-8a4a-ea71ca5e539d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz\" (UID: \"df543b0d-1120-499b-8a4a-ea71ca5e539d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz" Sep 10 15:47:57.310844 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.310662 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df543b0d-1120-499b-8a4a-ea71ca5e539d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz\" (UID: \"df543b0d-1120-499b-8a4a-ea71ca5e539d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz" Sep 10 15:47:57.311038 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.310995 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc964sf\" (UID: \"ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc964sf" Sep 10 15:47:57.311081 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.311042 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc964sf\" (UID: \"ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc964sf" Sep 10 15:47:57.311081 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.311057 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc964sf\" (UID: \"ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc964sf" Sep 10 15:47:57.311401 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.311379 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc964sf\" (UID: \"ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc964sf" Sep 10 15:47:57.312635 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.312614 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc964sf\" (UID: \"ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc964sf" Sep 10 15:47:57.313024 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.313005 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc964sf\" (UID: \"ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc964sf" Sep 10 15:47:57.319150 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.319127 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc964sf\" (UID: \"ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc964sf" Sep 10 15:47:57.319418 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.319401 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv2ww\" (UniqueName: \"kubernetes.io/projected/ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076-kube-api-access-tv2ww\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc964sf\" (UID: \"ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc964sf" Sep 10 15:47:57.380755 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.380721 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:57.411873 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.411814 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df543b0d-1120-499b-8a4a-ea71ca5e539d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz\" (UID: \"df543b0d-1120-499b-8a4a-ea71ca5e539d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz" Sep 10 15:47:57.411873 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.411855 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljftf\" (UniqueName: \"kubernetes.io/projected/df543b0d-1120-499b-8a4a-ea71ca5e539d-kube-api-access-ljftf\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz\" (UID: \"df543b0d-1120-499b-8a4a-ea71ca5e539d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz" Sep 10 15:47:57.412002 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.411888 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df543b0d-1120-499b-8a4a-ea71ca5e539d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz\" (UID: \"df543b0d-1120-499b-8a4a-ea71ca5e539d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz" Sep 10 15:47:57.412002 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.411911 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df543b0d-1120-499b-8a4a-ea71ca5e539d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz\" (UID: \"df543b0d-1120-499b-8a4a-ea71ca5e539d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz" Sep 10 15:47:57.412002 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.411965 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df543b0d-1120-499b-8a4a-ea71ca5e539d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz\" (UID: \"df543b0d-1120-499b-8a4a-ea71ca5e539d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz" Sep 10 15:47:57.412002 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.411990 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df543b0d-1120-499b-8a4a-ea71ca5e539d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz\" (UID: \"df543b0d-1120-499b-8a4a-ea71ca5e539d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz" Sep 10 15:47:57.412199 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.412022 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df543b0d-1120-499b-8a4a-ea71ca5e539d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz\" (UID: \"df543b0d-1120-499b-8a4a-ea71ca5e539d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz" Sep 10 15:47:57.412199 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.412062 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df543b0d-1120-499b-8a4a-ea71ca5e539d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz\" (UID: \"df543b0d-1120-499b-8a4a-ea71ca5e539d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz" Sep 10 15:47:57.412199 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.412104 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df543b0d-1120-499b-8a4a-ea71ca5e539d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz\" (UID: \"df543b0d-1120-499b-8a4a-ea71ca5e539d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz" Sep 10 15:47:57.412348 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.412193 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df543b0d-1120-499b-8a4a-ea71ca5e539d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz\" (UID: \"df543b0d-1120-499b-8a4a-ea71ca5e539d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz" Sep 10 15:47:57.412348 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.412261 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df543b0d-1120-499b-8a4a-ea71ca5e539d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz\" (UID: \"df543b0d-1120-499b-8a4a-ea71ca5e539d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz" Sep 10 15:47:57.412447 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.412414 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df543b0d-1120-499b-8a4a-ea71ca5e539d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz\" (UID: \"df543b0d-1120-499b-8a4a-ea71ca5e539d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz" Sep 10 15:47:57.412447 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.412427 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df543b0d-1120-499b-8a4a-ea71ca5e539d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz\" (UID: \"df543b0d-1120-499b-8a4a-ea71ca5e539d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz" Sep 10 15:47:57.412691 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.412675 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df543b0d-1120-499b-8a4a-ea71ca5e539d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz\" (UID: \"df543b0d-1120-499b-8a4a-ea71ca5e539d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz" Sep 10 15:47:57.414447 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.414431 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df543b0d-1120-499b-8a4a-ea71ca5e539d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz\" (UID: \"df543b0d-1120-499b-8a4a-ea71ca5e539d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz" Sep 10 15:47:57.414756 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.414719 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df543b0d-1120-499b-8a4a-ea71ca5e539d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz\" (UID: \"df543b0d-1120-499b-8a4a-ea71ca5e539d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz" Sep 10 15:47:57.420175 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.420154 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df543b0d-1120-499b-8a4a-ea71ca5e539d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz\" (UID: \"df543b0d-1120-499b-8a4a-ea71ca5e539d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz" Sep 10 15:47:57.420274 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.420162 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljftf\" (UniqueName: \"kubernetes.io/projected/df543b0d-1120-499b-8a4a-ea71ca5e539d-kube-api-access-ljftf\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz\" (UID: \"df543b0d-1120-499b-8a4a-ea71ca5e539d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz" Sep 10 15:47:57.435866 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.435849 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc964sf" Sep 10 15:47:57.524098 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.524066 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz" Sep 10 15:47:57.709171 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.709145 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc964sf"] Sep 10 15:47:57.711037 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:47:57.711004 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac2d5bdb_ced4_4ecb_9bbb_bf463f94a076.slice/crio-8eced91ece871fac30f09ece8433366cf70489d65833bc12e3b57905ca3ec1f7 WatchSource:0}: Error finding container 8eced91ece871fac30f09ece8433366cf70489d65833bc12e3b57905ca3ec1f7: Status 404 returned error can't find the container with id 8eced91ece871fac30f09ece8433366cf70489d65833bc12e3b57905ca3ec1f7 Sep 10 15:47:57.765237 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.765208 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc964sf" event={"ID":"ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076","Type":"ContainerStarted","Data":"8eced91ece871fac30f09ece8433366cf70489d65833bc12e3b57905ca3ec1f7"} Sep 10 15:47:57.809889 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:57.809871 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz"] Sep 10 15:47:57.811860 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:47:57.811838 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf543b0d_1120_499b_8a4a_ea71ca5e539d.slice/crio-0503f29e01b685fe43e0f92561777c0ee7b36f97e23b935c9898db2ca0558e0a WatchSource:0}: Error finding container 0503f29e01b685fe43e0f92561777c0ee7b36f97e23b935c9898db2ca0558e0a: Status 404 returned error can't find the container with id 0503f29e01b685fe43e0f92561777c0ee7b36f97e23b935c9898db2ca0558e0a Sep 10 15:47:58.772488 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:47:58.772445 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz" event={"ID":"df543b0d-1120-499b-8a4a-ea71ca5e539d","Type":"ContainerStarted","Data":"0503f29e01b685fe43e0f92561777c0ee7b36f97e23b935c9898db2ca0558e0a"} Sep 10 15:48:00.368061 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:00.368016 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 15:48:00.368495 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:00.368108 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 15:48:00.368495 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:00.368154 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 15:48:00.375231 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:00.375197 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 15:48:00.375323 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:00.375282 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 15:48:00.375364 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:00.375330 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 15:48:00.782933 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:00.782858 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc964sf" event={"ID":"ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076","Type":"ContainerStarted","Data":"216db2c75a62b09b5768e5f5cdbbc480fa93435f3fca7ff888813d53d7e7fe05"} Sep 10 15:48:00.784461 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:00.784437 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz" event={"ID":"df543b0d-1120-499b-8a4a-ea71ca5e539d","Type":"ContainerStarted","Data":"2e38ea95319277fe479b2e6446400c07b7df3fe0d06039a53f082cc00e93a5e7"} Sep 10 15:48:00.803446 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:00.803405 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc964sf" podStartSLOduration=1.141543203 podStartE2EDuration="3.803391832s" totalImagesPullingTime="2.661848629s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:47:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:48:00.802345017 +0000 UTC m=+358.155864217" watchObservedRunningTime="2026-09-10 15:48:00.803391832 +0000 UTC m=+358.156910993" Sep 10 15:48:00.823988 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:00.823949 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz" podStartSLOduration=1.269757073 podStartE2EDuration="3.823936607s" totalImagesPullingTime="2.554179534s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:47:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:48:00.822110145 +0000 UTC m=+358.175629319" watchObservedRunningTime="2026-09-10 15:48:00.823936607 +0000 UTC m=+358.177455768" Sep 10 15:48:01.283165 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:01.283122 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-8h7gh"] Sep 10 15:48:01.287707 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:01.287681 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-8h7gh" Sep 10 15:48:01.293164 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:01.293129 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-9plhr\"" Sep 10 15:48:01.293164 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:01.293163 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Sep 10 15:48:01.304670 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:01.304649 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-8h7gh"] Sep 10 15:48:01.346924 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:01.346895 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmhtj\" (UniqueName: \"kubernetes.io/projected/fc2d768d-cf98-4412-a63c-1ad1c466cdc5-kube-api-access-nmhtj\") pod \"odh-model-controller-7998c67dc4-8h7gh\" (UID: \"fc2d768d-cf98-4412-a63c-1ad1c466cdc5\") " pod="opendatahub/odh-model-controller-7998c67dc4-8h7gh" Sep 10 15:48:01.347036 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:01.346936 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fc2d768d-cf98-4412-a63c-1ad1c466cdc5-cert\") pod \"odh-model-controller-7998c67dc4-8h7gh\" (UID: \"fc2d768d-cf98-4412-a63c-1ad1c466cdc5\") " pod="opendatahub/odh-model-controller-7998c67dc4-8h7gh" Sep 10 15:48:01.363784 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:01.363754 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:48:01.437005 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:01.436974 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc964sf" Sep 10 15:48:01.438120 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:01.438091 2576 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bc964sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body= Sep 10 15:48:01.438219 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:01.438146 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc964sf" podUID="ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" Sep 10 15:48:01.448330 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:01.448307 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmhtj\" (UniqueName: \"kubernetes.io/projected/fc2d768d-cf98-4412-a63c-1ad1c466cdc5-kube-api-access-nmhtj\") pod \"odh-model-controller-7998c67dc4-8h7gh\" (UID: \"fc2d768d-cf98-4412-a63c-1ad1c466cdc5\") " pod="opendatahub/odh-model-controller-7998c67dc4-8h7gh" Sep 10 15:48:01.448423 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:01.448359 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fc2d768d-cf98-4412-a63c-1ad1c466cdc5-cert\") pod \"odh-model-controller-7998c67dc4-8h7gh\" (UID: \"fc2d768d-cf98-4412-a63c-1ad1c466cdc5\") " pod="opendatahub/odh-model-controller-7998c67dc4-8h7gh" Sep 10 15:48:01.448513 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:48:01.448497 2576 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Sep 10 15:48:01.448573 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:48:01.448563 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc2d768d-cf98-4412-a63c-1ad1c466cdc5-cert podName:fc2d768d-cf98-4412-a63c-1ad1c466cdc5 nodeName:}" failed. No retries permitted until 2026-09-10 15:48:01.948547798 +0000 UTC m=+359.302066951 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/fc2d768d-cf98-4412-a63c-1ad1c466cdc5-cert") pod "odh-model-controller-7998c67dc4-8h7gh" (UID: "fc2d768d-cf98-4412-a63c-1ad1c466cdc5") : secret "odh-model-controller-webhook-cert" not found Sep 10 15:48:01.459057 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:01.459030 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmhtj\" (UniqueName: \"kubernetes.io/projected/fc2d768d-cf98-4412-a63c-1ad1c466cdc5-kube-api-access-nmhtj\") pod \"odh-model-controller-7998c67dc4-8h7gh\" (UID: \"fc2d768d-cf98-4412-a63c-1ad1c466cdc5\") " pod="opendatahub/odh-model-controller-7998c67dc4-8h7gh" Sep 10 15:48:01.460128 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:01.460099 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:48:01.525117 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:01.525089 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz" Sep 10 15:48:01.530131 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:01.530106 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz" Sep 10 15:48:01.556575 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:01.556547 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:48:01.617913 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:01.617878 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:48:01.788812 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:01.788787 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz" Sep 10 15:48:01.789848 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:01.789826 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz" Sep 10 15:48:01.812472 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:01.812392 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-pn66h"] Sep 10 15:48:01.815678 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:01.815658 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-pn66h" Sep 10 15:48:01.817950 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:01.817928 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-frxqz\"" Sep 10 15:48:01.818108 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:01.818087 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Sep 10 15:48:01.827792 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:01.827710 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:48:01.827792 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:01.827784 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-pn66h"] Sep 10 15:48:01.845657 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:01.845637 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc964sf"] Sep 10 15:48:01.852430 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:01.852404 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75f36c0a-f51b-4ee6-8917-f9153bfa9f2d-tls-certs\") pod \"model-serving-api-54d89bd5fc-pn66h\" (UID: \"75f36c0a-f51b-4ee6-8917-f9153bfa9f2d\") " pod="opendatahub/model-serving-api-54d89bd5fc-pn66h" Sep 10 15:48:01.852524 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:01.852469 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22d9f\" (UniqueName: \"kubernetes.io/projected/75f36c0a-f51b-4ee6-8917-f9153bfa9f2d-kube-api-access-22d9f\") pod \"model-serving-api-54d89bd5fc-pn66h\" (UID: \"75f36c0a-f51b-4ee6-8917-f9153bfa9f2d\") " pod="opendatahub/model-serving-api-54d89bd5fc-pn66h" Sep 10 15:48:01.953329 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:01.953308 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fc2d768d-cf98-4412-a63c-1ad1c466cdc5-cert\") pod \"odh-model-controller-7998c67dc4-8h7gh\" (UID: \"fc2d768d-cf98-4412-a63c-1ad1c466cdc5\") " pod="opendatahub/odh-model-controller-7998c67dc4-8h7gh" Sep 10 15:48:01.953440 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:01.953357 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75f36c0a-f51b-4ee6-8917-f9153bfa9f2d-tls-certs\") pod \"model-serving-api-54d89bd5fc-pn66h\" (UID: \"75f36c0a-f51b-4ee6-8917-f9153bfa9f2d\") " pod="opendatahub/model-serving-api-54d89bd5fc-pn66h" Sep 10 15:48:01.953440 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:01.953387 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22d9f\" (UniqueName: \"kubernetes.io/projected/75f36c0a-f51b-4ee6-8917-f9153bfa9f2d-kube-api-access-22d9f\") pod \"model-serving-api-54d89bd5fc-pn66h\" (UID: \"75f36c0a-f51b-4ee6-8917-f9153bfa9f2d\") " pod="opendatahub/model-serving-api-54d89bd5fc-pn66h" Sep 10 15:48:01.955679 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:01.955657 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fc2d768d-cf98-4412-a63c-1ad1c466cdc5-cert\") pod \"odh-model-controller-7998c67dc4-8h7gh\" (UID: \"fc2d768d-cf98-4412-a63c-1ad1c466cdc5\") " pod="opendatahub/odh-model-controller-7998c67dc4-8h7gh" Sep 10 15:48:01.955855 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:01.955837 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75f36c0a-f51b-4ee6-8917-f9153bfa9f2d-tls-certs\") pod \"model-serving-api-54d89bd5fc-pn66h\" (UID: \"75f36c0a-f51b-4ee6-8917-f9153bfa9f2d\") " pod="opendatahub/model-serving-api-54d89bd5fc-pn66h" Sep 10 15:48:01.966265 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:01.966240 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22d9f\" (UniqueName: \"kubernetes.io/projected/75f36c0a-f51b-4ee6-8917-f9153bfa9f2d-kube-api-access-22d9f\") pod \"model-serving-api-54d89bd5fc-pn66h\" (UID: \"75f36c0a-f51b-4ee6-8917-f9153bfa9f2d\") " pod="opendatahub/model-serving-api-54d89bd5fc-pn66h" Sep 10 15:48:01.993745 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:01.993701 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:48:02.129335 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:02.129286 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-pn66h" Sep 10 15:48:02.198085 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:02.198057 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-8h7gh" Sep 10 15:48:02.394868 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:02.394845 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-pn66h"] Sep 10 15:48:02.396888 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:48:02.396861 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75f36c0a_f51b_4ee6_8917_f9153bfa9f2d.slice/crio-2c90185ed73d524e62064b07538bd503bcb995d377e20d9ff56bdfcd588bd2e1 WatchSource:0}: Error finding container 2c90185ed73d524e62064b07538bd503bcb995d377e20d9ff56bdfcd588bd2e1: Status 404 returned error can't find the container with id 2c90185ed73d524e62064b07538bd503bcb995d377e20d9ff56bdfcd588bd2e1 Sep 10 15:48:02.436778 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:02.436755 2576 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bc964sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body= Sep 10 15:48:02.436883 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:02.436808 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc964sf" podUID="ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" Sep 10 15:48:02.467659 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:02.467627 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-8h7gh"] Sep 10 15:48:02.469547 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:48:02.469524 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc2d768d_cf98_4412_a63c_1ad1c466cdc5.slice/crio-749a536da22e921a4b082971cd68e4fb0475236eb000ea425a1d8d8c67c5cbca WatchSource:0}: Error finding container 749a536da22e921a4b082971cd68e4fb0475236eb000ea425a1d8d8c67c5cbca: Status 404 returned error can't find the container with id 749a536da22e921a4b082971cd68e4fb0475236eb000ea425a1d8d8c67c5cbca Sep 10 15:48:02.794793 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:02.794758 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-pn66h" event={"ID":"75f36c0a-f51b-4ee6-8917-f9153bfa9f2d","Type":"ContainerStarted","Data":"2c90185ed73d524e62064b07538bd503bcb995d377e20d9ff56bdfcd588bd2e1"} Sep 10 15:48:02.795946 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:02.795920 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-8h7gh" event={"ID":"fc2d768d-cf98-4412-a63c-1ad1c466cdc5","Type":"ContainerStarted","Data":"749a536da22e921a4b082971cd68e4fb0475236eb000ea425a1d8d8c67c5cbca"} Sep 10 15:48:03.437631 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:03.437222 2576 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bc964sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body= Sep 10 15:48:03.437631 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:03.437281 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc964sf" podUID="ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" Sep 10 15:48:03.800807 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:03.800768 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc964sf" podUID="ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076" containerName="istio-proxy" containerID="cri-o://216db2c75a62b09b5768e5f5cdbbc480fa93435f3fca7ff888813d53d7e7fe05" gracePeriod=30 Sep 10 15:48:04.807663 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:04.807620 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-pn66h" event={"ID":"75f36c0a-f51b-4ee6-8917-f9153bfa9f2d","Type":"ContainerStarted","Data":"9c23cf8e078adc2c6326a7ba0dbdc12a616671b46f2da33dffa3a3a1816ce47d"} Sep 10 15:48:04.808141 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:04.807769 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-pn66h" Sep 10 15:48:04.825893 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:04.825841 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-pn66h" podStartSLOduration=2.410830171 podStartE2EDuration="3.825825947s" totalImagesPullingTime="1.414995776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:48:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:48:04.824104795 +0000 UTC m=+362.177623968" watchObservedRunningTime="2026-09-10 15:48:04.825825947 +0000 UTC m=+362.179345107" Sep 10 15:48:05.813696 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:05.813662 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-8h7gh" event={"ID":"fc2d768d-cf98-4412-a63c-1ad1c466cdc5","Type":"ContainerStarted","Data":"365cb3660ce2bee6039e3de0ff2219de63b0ed7fd7233397e3eb6750be76083c"} Sep 10 15:48:05.814113 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:05.813993 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-8h7gh" Sep 10 15:48:05.830509 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:05.830466 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-8h7gh" podStartSLOduration=2.126527415 podStartE2EDuration="4.830451688s" totalImagesPullingTime="2.703924273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:48:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:48:05.829713503 +0000 UTC m=+363.183232684" watchObservedRunningTime="2026-09-10 15:48:05.830451688 +0000 UTC m=+363.183970903" Sep 10 15:48:08.415502 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:08.415468 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-4xws7"] Sep 10 15:48:08.418637 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:08.418621 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4xws7" Sep 10 15:48:08.421804 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:08.421774 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 10 15:48:08.421919 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:08.421827 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 10 15:48:08.422068 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:08.422052 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 10 15:48:08.422116 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:08.422074 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-n7xm2\"" Sep 10 15:48:08.432929 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:08.432909 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-4xws7"] Sep 10 15:48:08.515486 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:08.515456 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/eff52c89-238c-4887-b268-2bf69a3fabab-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-4xws7\" (UID: \"eff52c89-238c-4887-b268-2bf69a3fabab\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4xws7" Sep 10 15:48:08.515619 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:08.515490 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/eff52c89-238c-4887-b268-2bf69a3fabab-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-4xws7\" (UID: \"eff52c89-238c-4887-b268-2bf69a3fabab\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4xws7" Sep 10 15:48:08.515619 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:08.515519 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/eff52c89-238c-4887-b268-2bf69a3fabab-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-4xws7\" (UID: \"eff52c89-238c-4887-b268-2bf69a3fabab\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4xws7" Sep 10 15:48:08.515619 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:08.515573 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eff52c89-238c-4887-b268-2bf69a3fabab-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-4xws7\" (UID: \"eff52c89-238c-4887-b268-2bf69a3fabab\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4xws7" Sep 10 15:48:08.515619 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:08.515605 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/eff52c89-238c-4887-b268-2bf69a3fabab-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-4xws7\" (UID: \"eff52c89-238c-4887-b268-2bf69a3fabab\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4xws7" Sep 10 15:48:08.515803 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:08.515629 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/eff52c89-238c-4887-b268-2bf69a3fabab-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-4xws7\" (UID: \"eff52c89-238c-4887-b268-2bf69a3fabab\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4xws7" Sep 10 15:48:08.515803 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:08.515646 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95ntj\" (UniqueName: \"kubernetes.io/projected/eff52c89-238c-4887-b268-2bf69a3fabab-kube-api-access-95ntj\") pod \"istiod-openshift-gateway-5677ccb49b-4xws7\" (UID: \"eff52c89-238c-4887-b268-2bf69a3fabab\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4xws7" Sep 10 15:48:08.616381 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:08.616353 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/eff52c89-238c-4887-b268-2bf69a3fabab-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-4xws7\" (UID: \"eff52c89-238c-4887-b268-2bf69a3fabab\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4xws7" Sep 10 15:48:08.616496 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:08.616389 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/eff52c89-238c-4887-b268-2bf69a3fabab-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-4xws7\" (UID: \"eff52c89-238c-4887-b268-2bf69a3fabab\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4xws7" Sep 10 15:48:08.616496 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:08.616417 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/eff52c89-238c-4887-b268-2bf69a3fabab-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-4xws7\" (UID: \"eff52c89-238c-4887-b268-2bf69a3fabab\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4xws7" Sep 10 15:48:08.616496 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:08.616456 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eff52c89-238c-4887-b268-2bf69a3fabab-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-4xws7\" (UID: \"eff52c89-238c-4887-b268-2bf69a3fabab\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4xws7" Sep 10 15:48:08.616636 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:08.616495 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/eff52c89-238c-4887-b268-2bf69a3fabab-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-4xws7\" (UID: \"eff52c89-238c-4887-b268-2bf69a3fabab\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4xws7" Sep 10 15:48:08.616636 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:08.616531 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/eff52c89-238c-4887-b268-2bf69a3fabab-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-4xws7\" (UID: \"eff52c89-238c-4887-b268-2bf69a3fabab\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4xws7" Sep 10 15:48:08.616636 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:08.616628 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95ntj\" (UniqueName: \"kubernetes.io/projected/eff52c89-238c-4887-b268-2bf69a3fabab-kube-api-access-95ntj\") pod \"istiod-openshift-gateway-5677ccb49b-4xws7\" (UID: \"eff52c89-238c-4887-b268-2bf69a3fabab\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4xws7" Sep 10 15:48:08.617520 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:08.617495 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/eff52c89-238c-4887-b268-2bf69a3fabab-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-4xws7\" (UID: \"eff52c89-238c-4887-b268-2bf69a3fabab\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4xws7" Sep 10 15:48:08.619023 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:08.618997 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/eff52c89-238c-4887-b268-2bf69a3fabab-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-4xws7\" (UID: \"eff52c89-238c-4887-b268-2bf69a3fabab\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4xws7" Sep 10 15:48:08.619108 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:08.619037 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/eff52c89-238c-4887-b268-2bf69a3fabab-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-4xws7\" (UID: \"eff52c89-238c-4887-b268-2bf69a3fabab\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4xws7" Sep 10 15:48:08.619108 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:08.619062 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/eff52c89-238c-4887-b268-2bf69a3fabab-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-4xws7\" (UID: \"eff52c89-238c-4887-b268-2bf69a3fabab\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4xws7" Sep 10 15:48:08.619108 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:08.619045 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/eff52c89-238c-4887-b268-2bf69a3fabab-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-4xws7\" (UID: \"eff52c89-238c-4887-b268-2bf69a3fabab\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4xws7" Sep 10 15:48:08.638059 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:08.638040 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95ntj\" (UniqueName: \"kubernetes.io/projected/eff52c89-238c-4887-b268-2bf69a3fabab-kube-api-access-95ntj\") pod \"istiod-openshift-gateway-5677ccb49b-4xws7\" (UID: \"eff52c89-238c-4887-b268-2bf69a3fabab\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4xws7" Sep 10 15:48:08.641060 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:08.641040 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eff52c89-238c-4887-b268-2bf69a3fabab-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-4xws7\" (UID: \"eff52c89-238c-4887-b268-2bf69a3fabab\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4xws7" Sep 10 15:48:08.728331 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:08.728282 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4xws7" Sep 10 15:48:09.008142 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:09.008071 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-4xws7"] Sep 10 15:48:09.011706 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:48:09.011683 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeff52c89_238c_4887_b268_2bf69a3fabab.slice/crio-0ed044cc9cb05ceccad4ce71d6b3a5e824ea6ebe0e18b9264c21fc9b0cc639f5 WatchSource:0}: Error finding container 0ed044cc9cb05ceccad4ce71d6b3a5e824ea6ebe0e18b9264c21fc9b0cc639f5: Status 404 returned error can't find the container with id 0ed044cc9cb05ceccad4ce71d6b3a5e824ea6ebe0e18b9264c21fc9b0cc639f5 Sep 10 15:48:09.038839 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:09.038801 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc964sf" Sep 10 15:48:09.120299 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:09.120275 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tv2ww\" (UniqueName: \"kubernetes.io/projected/ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076-kube-api-access-tv2ww\") pod \"ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076\" (UID: \"ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076\") " Sep 10 15:48:09.120393 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:09.120302 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076-istio-data\") pod \"ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076\" (UID: \"ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076\") " Sep 10 15:48:09.120393 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:09.120327 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076-istio-podinfo\") pod \"ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076\" (UID: \"ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076\") " Sep 10 15:48:09.120393 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:09.120351 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076-istiod-ca-cert\") pod \"ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076\" (UID: \"ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076\") " Sep 10 15:48:09.120393 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:09.120375 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076-workload-certs\") pod \"ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076\" (UID: \"ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076\") " Sep 10 15:48:09.120565 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:09.120421 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076-workload-socket\") pod \"ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076\" (UID: \"ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076\") " Sep 10 15:48:09.120565 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:09.120491 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076-istio-token\") pod \"ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076\" (UID: \"ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076\") " Sep 10 15:48:09.120565 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:09.120522 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076-credential-socket\") pod \"ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076\" (UID: \"ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076\") " Sep 10 15:48:09.120565 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:09.120552 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076-istio-envoy\") pod \"ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076\" (UID: \"ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076\") " Sep 10 15:48:09.120791 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:09.120637 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076-istio-data" (OuterVolumeSpecName: "istio-data") pod "ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076" (UID: "ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:48:09.120791 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:09.120649 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076" (UID: "ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:48:09.120902 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:09.120806 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076" (UID: "ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:48:09.120902 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:09.120873 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076-istio-data\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:48:09.120902 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:09.120851 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076" (UID: "ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:48:09.121265 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:09.120915 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076-workload-certs\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:48:09.121265 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:09.121009 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076" (UID: "ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:48:09.123275 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:09.123257 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076" (UID: "ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 15:48:09.123354 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:09.123283 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076-kube-api-access-tv2ww" (OuterVolumeSpecName: "kube-api-access-tv2ww") pod "ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076" (UID: "ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076"). InnerVolumeSpecName "kube-api-access-tv2ww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:48:09.123354 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:09.123332 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076" (UID: "ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:48:09.123420 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:09.123366 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076-istio-token" (OuterVolumeSpecName: "istio-token") pod "ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076" (UID: "ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:48:09.222014 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:09.221993 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076-istio-token\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:48:09.222192 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:09.222139 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076-credential-socket\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:48:09.222192 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:09.222163 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076-istio-envoy\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:48:09.222192 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:09.222179 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tv2ww\" (UniqueName: \"kubernetes.io/projected/ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076-kube-api-access-tv2ww\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:48:09.222192 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:09.222194 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076-istio-podinfo\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:48:09.222365 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:09.222216 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076-istiod-ca-cert\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:48:09.222365 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:09.222232 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076-workload-socket\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:48:09.833683 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:09.833649 2576 generic.go:356] "Generic (PLEG): container finished" podID="ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076" containerID="216db2c75a62b09b5768e5f5cdbbc480fa93435f3fca7ff888813d53d7e7fe05" exitCode=0 Sep 10 15:48:09.834146 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:09.833711 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc964sf" Sep 10 15:48:09.834146 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:09.833762 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc964sf" event={"ID":"ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076","Type":"ContainerDied","Data":"216db2c75a62b09b5768e5f5cdbbc480fa93435f3fca7ff888813d53d7e7fe05"} Sep 10 15:48:09.834146 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:09.833824 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc964sf" event={"ID":"ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076","Type":"ContainerDied","Data":"8eced91ece871fac30f09ece8433366cf70489d65833bc12e3b57905ca3ec1f7"} Sep 10 15:48:09.834146 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:09.833858 2576 scope.go:117] "RemoveContainer" containerID="216db2c75a62b09b5768e5f5cdbbc480fa93435f3fca7ff888813d53d7e7fe05" Sep 10 15:48:09.835707 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:09.835659 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4xws7" event={"ID":"eff52c89-238c-4887-b268-2bf69a3fabab","Type":"ContainerStarted","Data":"0ed044cc9cb05ceccad4ce71d6b3a5e824ea6ebe0e18b9264c21fc9b0cc639f5"} Sep 10 15:48:09.843627 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:09.843608 2576 scope.go:117] "RemoveContainer" containerID="216db2c75a62b09b5768e5f5cdbbc480fa93435f3fca7ff888813d53d7e7fe05" Sep 10 15:48:09.843934 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:48:09.843916 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"216db2c75a62b09b5768e5f5cdbbc480fa93435f3fca7ff888813d53d7e7fe05\": container with ID starting with 216db2c75a62b09b5768e5f5cdbbc480fa93435f3fca7ff888813d53d7e7fe05 not found: ID does not exist" containerID="216db2c75a62b09b5768e5f5cdbbc480fa93435f3fca7ff888813d53d7e7fe05" Sep 10 15:48:09.843992 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:09.843942 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"216db2c75a62b09b5768e5f5cdbbc480fa93435f3fca7ff888813d53d7e7fe05"} err="failed to get container status \"216db2c75a62b09b5768e5f5cdbbc480fa93435f3fca7ff888813d53d7e7fe05\": rpc error: code = NotFound desc = could not find container \"216db2c75a62b09b5768e5f5cdbbc480fa93435f3fca7ff888813d53d7e7fe05\": container with ID starting with 216db2c75a62b09b5768e5f5cdbbc480fa93435f3fca7ff888813d53d7e7fe05 not found: ID does not exist" Sep 10 15:48:09.856432 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:09.856410 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc964sf"] Sep 10 15:48:09.873987 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:09.873938 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc964sf"] Sep 10 15:48:11.155900 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:11.155856 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 15:48:11.156266 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:11.155949 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 15:48:11.168511 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:11.168488 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076" path="/var/lib/kubelet/pods/ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076/volumes" Sep 10 15:48:11.397546 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:11.397509 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:48:11.487624 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:11.484374 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-n9wld"] Sep 10 15:48:11.487624 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:11.484887 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076" containerName="istio-proxy" Sep 10 15:48:11.487624 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:11.484906 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076" containerName="istio-proxy" Sep 10 15:48:11.487624 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:11.485013 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac2d5bdb-ced4-4ecb-9bbb-bf463f94a076" containerName="istio-proxy" Sep 10 15:48:11.488406 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:11.488378 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-n9wld" Sep 10 15:48:11.491536 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:11.491496 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Sep 10 15:48:11.493427 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:11.493405 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-jcxp6\"" Sep 10 15:48:11.508692 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:11.508670 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-n9wld"] Sep 10 15:48:11.517667 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:11.517632 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-w7jnk"] Sep 10 15:48:11.518254 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:11.518220 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:48:11.521470 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:11.521449 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-w7jnk" Sep 10 15:48:11.524028 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:11.523879 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Sep 10 15:48:11.530461 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:11.530438 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-t5jmw\"" Sep 10 15:48:11.533597 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:11.533575 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-w7jnk"] Sep 10 15:48:11.545857 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:11.545826 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da14561b-2f8c-4987-953f-260f78e954db-cert\") pod \"kserve-controller-manager-d6f4b65b8-n9wld\" (UID: \"da14561b-2f8c-4987-953f-260f78e954db\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-n9wld" Sep 10 15:48:11.546040 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:11.545960 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp7rz\" (UniqueName: \"kubernetes.io/projected/da14561b-2f8c-4987-953f-260f78e954db-kube-api-access-jp7rz\") pod \"kserve-controller-manager-d6f4b65b8-n9wld\" (UID: \"da14561b-2f8c-4987-953f-260f78e954db\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-n9wld" Sep 10 15:48:11.646553 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:11.646518 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da14561b-2f8c-4987-953f-260f78e954db-cert\") pod \"kserve-controller-manager-d6f4b65b8-n9wld\" (UID: \"da14561b-2f8c-4987-953f-260f78e954db\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-n9wld" Sep 10 15:48:11.646727 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:11.646590 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfqhb\" (UniqueName: \"kubernetes.io/projected/bfd89460-9cbb-4808-ad22-8b14f6e690ea-kube-api-access-bfqhb\") pod \"llmisvc-controller-manager-777c6bd7d5-w7jnk\" (UID: \"bfd89460-9cbb-4808-ad22-8b14f6e690ea\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-w7jnk" Sep 10 15:48:11.646727 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:11.646618 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bfd89460-9cbb-4808-ad22-8b14f6e690ea-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-w7jnk\" (UID: \"bfd89460-9cbb-4808-ad22-8b14f6e690ea\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-w7jnk" Sep 10 15:48:11.646727 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:11.646650 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp7rz\" (UniqueName: \"kubernetes.io/projected/da14561b-2f8c-4987-953f-260f78e954db-kube-api-access-jp7rz\") pod \"kserve-controller-manager-d6f4b65b8-n9wld\" (UID: \"da14561b-2f8c-4987-953f-260f78e954db\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-n9wld" Sep 10 15:48:11.646727 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:48:11.646673 2576 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Sep 10 15:48:11.646896 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:48:11.646781 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da14561b-2f8c-4987-953f-260f78e954db-cert podName:da14561b-2f8c-4987-953f-260f78e954db nodeName:}" failed. No retries permitted until 2026-09-10 15:48:12.14675764 +0000 UTC m=+369.500276793 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/da14561b-2f8c-4987-953f-260f78e954db-cert") pod "kserve-controller-manager-d6f4b65b8-n9wld" (UID: "da14561b-2f8c-4987-953f-260f78e954db") : secret "kserve-webhook-server-cert" not found Sep 10 15:48:11.656173 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:11.656143 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp7rz\" (UniqueName: \"kubernetes.io/projected/da14561b-2f8c-4987-953f-260f78e954db-kube-api-access-jp7rz\") pod \"kserve-controller-manager-d6f4b65b8-n9wld\" (UID: \"da14561b-2f8c-4987-953f-260f78e954db\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-n9wld" Sep 10 15:48:11.748010 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:11.747936 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfqhb\" (UniqueName: \"kubernetes.io/projected/bfd89460-9cbb-4808-ad22-8b14f6e690ea-kube-api-access-bfqhb\") pod \"llmisvc-controller-manager-777c6bd7d5-w7jnk\" (UID: \"bfd89460-9cbb-4808-ad22-8b14f6e690ea\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-w7jnk" Sep 10 15:48:11.748010 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:11.747974 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bfd89460-9cbb-4808-ad22-8b14f6e690ea-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-w7jnk\" (UID: \"bfd89460-9cbb-4808-ad22-8b14f6e690ea\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-w7jnk" Sep 10 15:48:11.748173 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:48:11.748075 2576 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Sep 10 15:48:11.748173 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:48:11.748122 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfd89460-9cbb-4808-ad22-8b14f6e690ea-cert podName:bfd89460-9cbb-4808-ad22-8b14f6e690ea nodeName:}" failed. No retries permitted until 2026-09-10 15:48:12.248109466 +0000 UTC m=+369.601628605 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/bfd89460-9cbb-4808-ad22-8b14f6e690ea-cert") pod "llmisvc-controller-manager-777c6bd7d5-w7jnk" (UID: "bfd89460-9cbb-4808-ad22-8b14f6e690ea") : secret "llmisvc-webhook-server-cert" not found Sep 10 15:48:11.765718 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:11.765682 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfqhb\" (UniqueName: \"kubernetes.io/projected/bfd89460-9cbb-4808-ad22-8b14f6e690ea-kube-api-access-bfqhb\") pod \"llmisvc-controller-manager-777c6bd7d5-w7jnk\" (UID: \"bfd89460-9cbb-4808-ad22-8b14f6e690ea\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-w7jnk" Sep 10 15:48:11.787189 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:11.787162 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:48:11.849590 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:11.849566 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4xws7" event={"ID":"eff52c89-238c-4887-b268-2bf69a3fabab","Type":"ContainerStarted","Data":"85563d6f48d1638793b20f86f0257c70cb92157ba04ce412ea6865635169eb5d"} Sep 10 15:48:11.849705 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:11.849690 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4xws7" Sep 10 15:48:11.897079 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:11.897034 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4xws7" podStartSLOduration=1.755223543 podStartE2EDuration="3.897020997s" totalImagesPullingTime="2.141797454s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:48:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:48:11.895002716 +0000 UTC m=+369.248521895" watchObservedRunningTime="2026-09-10 15:48:11.897020997 +0000 UTC m=+369.250540158" Sep 10 15:48:12.151724 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:12.151695 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da14561b-2f8c-4987-953f-260f78e954db-cert\") pod \"kserve-controller-manager-d6f4b65b8-n9wld\" (UID: \"da14561b-2f8c-4987-953f-260f78e954db\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-n9wld" Sep 10 15:48:12.154234 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:12.154212 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da14561b-2f8c-4987-953f-260f78e954db-cert\") pod \"kserve-controller-manager-d6f4b65b8-n9wld\" (UID: \"da14561b-2f8c-4987-953f-260f78e954db\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-n9wld" Sep 10 15:48:12.252995 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:12.252965 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bfd89460-9cbb-4808-ad22-8b14f6e690ea-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-w7jnk\" (UID: \"bfd89460-9cbb-4808-ad22-8b14f6e690ea\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-w7jnk" Sep 10 15:48:12.255396 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:12.255377 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bfd89460-9cbb-4808-ad22-8b14f6e690ea-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-w7jnk\" (UID: \"bfd89460-9cbb-4808-ad22-8b14f6e690ea\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-w7jnk" Sep 10 15:48:12.273499 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:12.273464 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:48:12.400478 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:12.400443 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-n9wld" Sep 10 15:48:12.435027 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:12.434997 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-w7jnk" Sep 10 15:48:12.695779 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:12.695682 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-n9wld"] Sep 10 15:48:12.702634 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:48:12.702607 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda14561b_2f8c_4987_953f_260f78e954db.slice/crio-f5b45ce333fb1e0bb3101f382797697c163c237ee907be43cb9ee660d9384757 WatchSource:0}: Error finding container f5b45ce333fb1e0bb3101f382797697c163c237ee907be43cb9ee660d9384757: Status 404 returned error can't find the container with id f5b45ce333fb1e0bb3101f382797697c163c237ee907be43cb9ee660d9384757 Sep 10 15:48:12.737653 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:12.737540 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-w7jnk"] Sep 10 15:48:12.739967 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:48:12.739942 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbfd89460_9cbb_4808_ad22_8b14f6e690ea.slice/crio-684cab2395d3e6d7de497f991eeea5ce20b8c14ef8212555af4f3fe849147645 WatchSource:0}: Error finding container 684cab2395d3e6d7de497f991eeea5ce20b8c14ef8212555af4f3fe849147645: Status 404 returned error can't find the container with id 684cab2395d3e6d7de497f991eeea5ce20b8c14ef8212555af4f3fe849147645 Sep 10 15:48:12.747194 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:12.747166 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:48:12.837890 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:12.837852 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:48:12.854650 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:12.854622 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-w7jnk" event={"ID":"bfd89460-9cbb-4808-ad22-8b14f6e690ea","Type":"ContainerStarted","Data":"684cab2395d3e6d7de497f991eeea5ce20b8c14ef8212555af4f3fe849147645"} Sep 10 15:48:12.855725 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:12.855702 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-n9wld" event={"ID":"da14561b-2f8c-4987-953f-260f78e954db","Type":"ContainerStarted","Data":"f5b45ce333fb1e0bb3101f382797697c163c237ee907be43cb9ee660d9384757"} Sep 10 15:48:12.857632 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:12.857609 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4xws7" Sep 10 15:48:13.001248 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:13.001165 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:48:15.819419 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:15.819391 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-pn66h" Sep 10 15:48:15.878186 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:15.878152 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-w7jnk" event={"ID":"bfd89460-9cbb-4808-ad22-8b14f6e690ea","Type":"ContainerStarted","Data":"f5bebe41b24cd85d6304e67b1bd406ce17a89401c9ca714d3ceb96952ee3e599"} Sep 10 15:48:15.878360 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:15.878286 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-w7jnk" Sep 10 15:48:15.879756 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:15.879716 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-n9wld" event={"ID":"da14561b-2f8c-4987-953f-260f78e954db","Type":"ContainerStarted","Data":"bbd1a4f82c07cf851bfb24aec7a6f6cd3c6d57ab7e22ab722675042cdd5c2955"} Sep 10 15:48:15.879873 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:15.879781 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-n9wld" Sep 10 15:48:15.896108 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:15.896067 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-w7jnk" podStartSLOduration=2.103185722 podStartE2EDuration="4.896054329s" totalImagesPullingTime="2.792868607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:48:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:48:15.89494063 +0000 UTC m=+373.248459793" watchObservedRunningTime="2026-09-10 15:48:15.896054329 +0000 UTC m=+373.249573488" Sep 10 15:48:15.915542 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:15.915501 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-n9wld" podStartSLOduration=2.690743764 podStartE2EDuration="4.915486268s" totalImagesPullingTime="2.224742504s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:48:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:48:15.913487324 +0000 UTC m=+373.267006504" watchObservedRunningTime="2026-09-10 15:48:15.915486268 +0000 UTC m=+373.269005429" Sep 10 15:48:16.819752 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:16.819711 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-8h7gh" Sep 10 15:48:31.734248 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:31.734213 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:48:31.791432 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:31.791397 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:48:46.886641 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:46.886612 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-w7jnk" Sep 10 15:48:46.889680 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:48:46.889661 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-n9wld" Sep 10 15:49:02.137905 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:02.137858 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:02.227552 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:02.227509 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:15.355691 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:15.355659 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:16.152172 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:16.152140 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:16.991523 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:16.991486 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:19.535396 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:19.535356 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:20.765530 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:20.765491 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:21.804283 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:21.804247 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:22.958264 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:22.958229 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:23.349129 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:23.349095 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:24.663451 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:24.663378 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-fs988"] Sep 10 15:49:24.667950 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:24.667933 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fs988" Sep 10 15:49:24.670942 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:24.670920 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 10 15:49:24.671035 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:24.671015 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-lgwqb\"" Sep 10 15:49:24.671718 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:24.671699 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 10 15:49:24.685169 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:24.685149 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-fs988"] Sep 10 15:49:24.757985 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:24.757950 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:24.768967 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:24.768937 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dp2k\" (UniqueName: \"kubernetes.io/projected/4e511f8e-7822-4387-a6ac-7a50e5e6350b-kube-api-access-7dp2k\") pod \"limitador-operator-controller-manager-74495c69f8-fs988\" (UID: \"4e511f8e-7822-4387-a6ac-7a50e5e6350b\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fs988" Sep 10 15:49:24.870217 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:24.870188 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dp2k\" (UniqueName: \"kubernetes.io/projected/4e511f8e-7822-4387-a6ac-7a50e5e6350b-kube-api-access-7dp2k\") pod \"limitador-operator-controller-manager-74495c69f8-fs988\" (UID: \"4e511f8e-7822-4387-a6ac-7a50e5e6350b\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fs988" Sep 10 15:49:24.882218 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:24.882191 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dp2k\" (UniqueName: \"kubernetes.io/projected/4e511f8e-7822-4387-a6ac-7a50e5e6350b-kube-api-access-7dp2k\") pod \"limitador-operator-controller-manager-74495c69f8-fs988\" (UID: \"4e511f8e-7822-4387-a6ac-7a50e5e6350b\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fs988" Sep 10 15:49:24.977855 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:24.977782 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fs988" Sep 10 15:49:25.140533 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:25.140503 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:25.257820 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:25.257745 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-fs988"] Sep 10 15:49:25.260897 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:49:25.260872 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e511f8e_7822_4387_a6ac_7a50e5e6350b.slice/crio-398e586a6a611f417aec656cd3c8c91cecc03259685702152206b0f9a21e652b WatchSource:0}: Error finding container 398e586a6a611f417aec656cd3c8c91cecc03259685702152206b0f9a21e652b: Status 404 returned error can't find the container with id 398e586a6a611f417aec656cd3c8c91cecc03259685702152206b0f9a21e652b Sep 10 15:49:26.193379 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:26.193340 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fs988" event={"ID":"4e511f8e-7822-4387-a6ac-7a50e5e6350b","Type":"ContainerStarted","Data":"398e586a6a611f417aec656cd3c8c91cecc03259685702152206b0f9a21e652b"} Sep 10 15:49:27.204330 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:27.204285 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:28.204974 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:28.204941 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fs988" event={"ID":"4e511f8e-7822-4387-a6ac-7a50e5e6350b","Type":"ContainerStarted","Data":"6d15c5ccdc5b674fb5a0ce38ddc5bc15fa366f8f89c454f3a16b92245200437f"} Sep 10 15:49:28.205345 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:28.205016 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fs988" Sep 10 15:49:28.254072 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:28.254016 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:28.306709 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:28.306656 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fs988" podStartSLOduration=2.019504785 podStartE2EDuration="4.306641048s" totalImagesPullingTime="2.287136263s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:49:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:49:28.305227486 +0000 UTC m=+445.658746669" watchObservedRunningTime="2026-09-10 15:49:28.306641048 +0000 UTC m=+445.660160209" Sep 10 15:49:28.499065 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:28.498984 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:30.578493 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:30.578457 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:30.661374 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:30.661343 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:30.947391 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:30.947295 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:31.046366 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:31.046333 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:31.440856 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:31.440808 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:31.518420 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:31.518387 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:37.009772 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:37.009711 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:37.150030 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:37.150000 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:38.393809 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:38.393778 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-fs988"] Sep 10 15:49:38.394234 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:38.393999 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fs988" podUID="4e511f8e-7822-4387-a6ac-7a50e5e6350b" containerName="manager" containerID="cri-o://6d15c5ccdc5b674fb5a0ce38ddc5bc15fa366f8f89c454f3a16b92245200437f" gracePeriod=2 Sep 10 15:49:38.396211 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:38.396180 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fs988" Sep 10 15:49:38.413485 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:38.413454 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-fs988"] Sep 10 15:49:38.427229 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:49:38.427198 2576 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"4e511f8e-7822-4387-a6ac-7a50e5e6350b\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-74495c69f8-fs988\": pods \"limitador-operator-controller-manager-74495c69f8-fs988\" not found" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fs988" Sep 10 15:49:38.431189 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:49:38.431153 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-fs988\" is forbidden: User \"system:node:ip-10-0-139-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-236.ec2.internal' and this object" podUID="4e511f8e-7822-4387-a6ac-7a50e5e6350b" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fs988" Sep 10 15:49:38.432762 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:38.432726 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-s25wn"] Sep 10 15:49:38.433092 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:38.433080 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e511f8e-7822-4387-a6ac-7a50e5e6350b" containerName="manager" Sep 10 15:49:38.433131 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:38.433094 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e511f8e-7822-4387-a6ac-7a50e5e6350b" containerName="manager" Sep 10 15:49:38.433164 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:38.433154 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e511f8e-7822-4387-a6ac-7a50e5e6350b" containerName="manager" Sep 10 15:49:38.436223 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:38.436206 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-s25wn" Sep 10 15:49:38.439359 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:49:38.439333 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-fs988\" is forbidden: User \"system:node:ip-10-0-139-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-236.ec2.internal' and this object" podUID="4e511f8e-7822-4387-a6ac-7a50e5e6350b" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fs988" Sep 10 15:49:38.458655 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:38.458629 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-s25wn"] Sep 10 15:49:38.487031 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:38.487002 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjjrr\" (UniqueName: \"kubernetes.io/projected/3a136beb-6a6c-420b-9d5a-87835dc85964-kube-api-access-cjjrr\") pod \"limitador-operator-controller-manager-74495c69f8-s25wn\" (UID: \"3a136beb-6a6c-420b-9d5a-87835dc85964\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-s25wn" Sep 10 15:49:38.588302 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:38.588263 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjjrr\" (UniqueName: \"kubernetes.io/projected/3a136beb-6a6c-420b-9d5a-87835dc85964-kube-api-access-cjjrr\") pod \"limitador-operator-controller-manager-74495c69f8-s25wn\" (UID: \"3a136beb-6a6c-420b-9d5a-87835dc85964\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-s25wn" Sep 10 15:49:38.604954 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:38.604925 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjjrr\" (UniqueName: \"kubernetes.io/projected/3a136beb-6a6c-420b-9d5a-87835dc85964-kube-api-access-cjjrr\") pod \"limitador-operator-controller-manager-74495c69f8-s25wn\" (UID: \"3a136beb-6a6c-420b-9d5a-87835dc85964\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-s25wn" Sep 10 15:49:38.637724 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:38.637700 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fs988" Sep 10 15:49:38.640053 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:49:38.640028 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-fs988\" is forbidden: User \"system:node:ip-10-0-139-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-236.ec2.internal' and this object" podUID="4e511f8e-7822-4387-a6ac-7a50e5e6350b" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fs988" Sep 10 15:49:38.689230 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:38.689183 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7dp2k\" (UniqueName: \"kubernetes.io/projected/4e511f8e-7822-4387-a6ac-7a50e5e6350b-kube-api-access-7dp2k\") pod \"4e511f8e-7822-4387-a6ac-7a50e5e6350b\" (UID: \"4e511f8e-7822-4387-a6ac-7a50e5e6350b\") " Sep 10 15:49:38.691389 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:38.691362 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e511f8e-7822-4387-a6ac-7a50e5e6350b-kube-api-access-7dp2k" (OuterVolumeSpecName: "kube-api-access-7dp2k") pod "4e511f8e-7822-4387-a6ac-7a50e5e6350b" (UID: "4e511f8e-7822-4387-a6ac-7a50e5e6350b"). InnerVolumeSpecName "kube-api-access-7dp2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:49:38.769237 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:38.769210 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-s25wn" Sep 10 15:49:38.790203 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:38.790178 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dp2k\" (UniqueName: \"kubernetes.io/projected/4e511f8e-7822-4387-a6ac-7a50e5e6350b-kube-api-access-7dp2k\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:49:39.053206 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:39.053115 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-s25wn"] Sep 10 15:49:39.056149 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:49:39.056123 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a136beb_6a6c_420b_9d5a_87835dc85964.slice/crio-c79df73a17968fb648dca6ee1eedba136dd07fc6e4effca65b02b0ac1ea7632a WatchSource:0}: Error finding container c79df73a17968fb648dca6ee1eedba136dd07fc6e4effca65b02b0ac1ea7632a: Status 404 returned error can't find the container with id c79df73a17968fb648dca6ee1eedba136dd07fc6e4effca65b02b0ac1ea7632a Sep 10 15:49:39.168187 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:39.168153 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e511f8e-7822-4387-a6ac-7a50e5e6350b" path="/var/lib/kubelet/pods/4e511f8e-7822-4387-a6ac-7a50e5e6350b/volumes" Sep 10 15:49:39.255745 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:39.255709 2576 generic.go:356] "Generic (PLEG): container finished" podID="4e511f8e-7822-4387-a6ac-7a50e5e6350b" containerID="6d15c5ccdc5b674fb5a0ce38ddc5bc15fa366f8f89c454f3a16b92245200437f" exitCode=0 Sep 10 15:49:39.255910 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:39.255777 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fs988" Sep 10 15:49:39.255910 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:39.255795 2576 scope.go:117] "RemoveContainer" containerID="6d15c5ccdc5b674fb5a0ce38ddc5bc15fa366f8f89c454f3a16b92245200437f" Sep 10 15:49:39.257618 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:39.257597 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-s25wn" event={"ID":"3a136beb-6a6c-420b-9d5a-87835dc85964","Type":"ContainerStarted","Data":"15ef8cea86ba907faa502149b7de8913f20d46b044262f610e2de3afd1f1eeb0"} Sep 10 15:49:39.257788 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:39.257643 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-s25wn" event={"ID":"3a136beb-6a6c-420b-9d5a-87835dc85964","Type":"ContainerStarted","Data":"c79df73a17968fb648dca6ee1eedba136dd07fc6e4effca65b02b0ac1ea7632a"} Sep 10 15:49:39.257788 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:39.257695 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-s25wn" Sep 10 15:49:39.263801 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:39.263784 2576 scope.go:117] "RemoveContainer" containerID="6d15c5ccdc5b674fb5a0ce38ddc5bc15fa366f8f89c454f3a16b92245200437f" Sep 10 15:49:39.264083 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:49:39.264053 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d15c5ccdc5b674fb5a0ce38ddc5bc15fa366f8f89c454f3a16b92245200437f\": container with ID starting with 6d15c5ccdc5b674fb5a0ce38ddc5bc15fa366f8f89c454f3a16b92245200437f not found: ID does not exist" containerID="6d15c5ccdc5b674fb5a0ce38ddc5bc15fa366f8f89c454f3a16b92245200437f" Sep 10 15:49:39.264147 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:39.264086 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d15c5ccdc5b674fb5a0ce38ddc5bc15fa366f8f89c454f3a16b92245200437f"} err="failed to get container status \"6d15c5ccdc5b674fb5a0ce38ddc5bc15fa366f8f89c454f3a16b92245200437f\": rpc error: code = NotFound desc = could not find container \"6d15c5ccdc5b674fb5a0ce38ddc5bc15fa366f8f89c454f3a16b92245200437f\": container with ID starting with 6d15c5ccdc5b674fb5a0ce38ddc5bc15fa366f8f89c454f3a16b92245200437f not found: ID does not exist" Sep 10 15:49:39.302809 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:39.302764 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-s25wn" podStartSLOduration=1.302749328 podStartE2EDuration="1.302749328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:49:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:49:39.301521889 +0000 UTC m=+456.655041068" watchObservedRunningTime="2026-09-10 15:49:39.302749328 +0000 UTC m=+456.656268493" Sep 10 15:49:41.650980 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:41.650949 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:41.755548 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:41.755511 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:46.400108 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:46.400070 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:46.491464 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:46.491430 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:48.896885 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:48.896850 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:49.005606 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:49.005563 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:50.265470 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:50.265439 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-s25wn" Sep 10 15:49:51.559548 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:51.559506 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:51.659343 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:51.659295 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:51.951286 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:51.951187 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:52.011230 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:52.011184 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:52.168815 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:52.168788 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:52.269693 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:52.269591 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:58.616046 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:58.616004 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:58.742071 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:49:58.742038 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:50:03.372924 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:03.372887 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:50:03.474131 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:03.474058 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:50:08.165977 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:08.165939 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:50:08.266515 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:08.266480 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:50:12.422851 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:12.422818 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fq9cv"] Sep 10 15:50:12.429476 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:12.429451 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fq9cv" Sep 10 15:50:12.432245 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:12.432225 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-m76tj\"" Sep 10 15:50:12.441942 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:12.441919 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fq9cv"] Sep 10 15:50:12.460129 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:12.460104 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:50:12.563900 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:12.563866 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b754538-f8b9-4891-aa4f-b910ddd9d35e-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fq9cv\" (UID: \"0b754538-f8b9-4891-aa4f-b910ddd9d35e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fq9cv" Sep 10 15:50:12.563900 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:12.563907 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b754538-f8b9-4891-aa4f-b910ddd9d35e-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fq9cv\" (UID: \"0b754538-f8b9-4891-aa4f-b910ddd9d35e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fq9cv" Sep 10 15:50:12.564123 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:12.563937 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b754538-f8b9-4891-aa4f-b910ddd9d35e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fq9cv\" (UID: \"0b754538-f8b9-4891-aa4f-b910ddd9d35e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fq9cv" Sep 10 15:50:12.564123 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:12.564041 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b754538-f8b9-4891-aa4f-b910ddd9d35e-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fq9cv\" (UID: \"0b754538-f8b9-4891-aa4f-b910ddd9d35e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fq9cv" Sep 10 15:50:12.564123 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:12.564069 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48cwk\" (UniqueName: \"kubernetes.io/projected/0b754538-f8b9-4891-aa4f-b910ddd9d35e-kube-api-access-48cwk\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fq9cv\" (UID: \"0b754538-f8b9-4891-aa4f-b910ddd9d35e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fq9cv" Sep 10 15:50:12.564225 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:12.564119 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b754538-f8b9-4891-aa4f-b910ddd9d35e-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fq9cv\" (UID: \"0b754538-f8b9-4891-aa4f-b910ddd9d35e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fq9cv" Sep 10 15:50:12.564225 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:12.564153 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b754538-f8b9-4891-aa4f-b910ddd9d35e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fq9cv\" (UID: \"0b754538-f8b9-4891-aa4f-b910ddd9d35e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fq9cv" Sep 10 15:50:12.564225 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:12.564214 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b754538-f8b9-4891-aa4f-b910ddd9d35e-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fq9cv\" (UID: \"0b754538-f8b9-4891-aa4f-b910ddd9d35e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fq9cv" Sep 10 15:50:12.564317 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:12.564248 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b754538-f8b9-4891-aa4f-b910ddd9d35e-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fq9cv\" (UID: \"0b754538-f8b9-4891-aa4f-b910ddd9d35e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fq9cv" Sep 10 15:50:12.665649 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:12.665617 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b754538-f8b9-4891-aa4f-b910ddd9d35e-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fq9cv\" (UID: \"0b754538-f8b9-4891-aa4f-b910ddd9d35e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fq9cv" Sep 10 15:50:12.665649 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:12.665653 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48cwk\" (UniqueName: \"kubernetes.io/projected/0b754538-f8b9-4891-aa4f-b910ddd9d35e-kube-api-access-48cwk\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fq9cv\" (UID: \"0b754538-f8b9-4891-aa4f-b910ddd9d35e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fq9cv" Sep 10 15:50:12.665899 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:12.665677 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b754538-f8b9-4891-aa4f-b910ddd9d35e-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fq9cv\" (UID: \"0b754538-f8b9-4891-aa4f-b910ddd9d35e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fq9cv" Sep 10 15:50:12.665899 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:12.665694 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b754538-f8b9-4891-aa4f-b910ddd9d35e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fq9cv\" (UID: \"0b754538-f8b9-4891-aa4f-b910ddd9d35e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fq9cv" Sep 10 15:50:12.665899 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:12.665825 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b754538-f8b9-4891-aa4f-b910ddd9d35e-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fq9cv\" (UID: \"0b754538-f8b9-4891-aa4f-b910ddd9d35e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fq9cv" Sep 10 15:50:12.665899 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:12.665875 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b754538-f8b9-4891-aa4f-b910ddd9d35e-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fq9cv\" (UID: \"0b754538-f8b9-4891-aa4f-b910ddd9d35e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fq9cv" Sep 10 15:50:12.666129 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:12.665981 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b754538-f8b9-4891-aa4f-b910ddd9d35e-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fq9cv\" (UID: \"0b754538-f8b9-4891-aa4f-b910ddd9d35e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fq9cv" Sep 10 15:50:12.666129 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:12.666006 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b754538-f8b9-4891-aa4f-b910ddd9d35e-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fq9cv\" (UID: \"0b754538-f8b9-4891-aa4f-b910ddd9d35e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fq9cv" Sep 10 15:50:12.666129 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:12.666033 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b754538-f8b9-4891-aa4f-b910ddd9d35e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fq9cv\" (UID: \"0b754538-f8b9-4891-aa4f-b910ddd9d35e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fq9cv" Sep 10 15:50:12.666129 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:12.666047 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b754538-f8b9-4891-aa4f-b910ddd9d35e-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fq9cv\" (UID: \"0b754538-f8b9-4891-aa4f-b910ddd9d35e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fq9cv" Sep 10 15:50:12.666348 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:12.666301 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b754538-f8b9-4891-aa4f-b910ddd9d35e-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fq9cv\" (UID: \"0b754538-f8b9-4891-aa4f-b910ddd9d35e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fq9cv" Sep 10 15:50:12.666403 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:12.666344 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b754538-f8b9-4891-aa4f-b910ddd9d35e-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fq9cv\" (UID: \"0b754538-f8b9-4891-aa4f-b910ddd9d35e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fq9cv" Sep 10 15:50:12.666626 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:12.666433 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b754538-f8b9-4891-aa4f-b910ddd9d35e-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fq9cv\" (UID: \"0b754538-f8b9-4891-aa4f-b910ddd9d35e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fq9cv" Sep 10 15:50:12.666626 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:12.666490 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b754538-f8b9-4891-aa4f-b910ddd9d35e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fq9cv\" (UID: \"0b754538-f8b9-4891-aa4f-b910ddd9d35e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fq9cv" Sep 10 15:50:12.668197 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:12.668175 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b754538-f8b9-4891-aa4f-b910ddd9d35e-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fq9cv\" (UID: \"0b754538-f8b9-4891-aa4f-b910ddd9d35e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fq9cv" Sep 10 15:50:12.668816 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:12.668800 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b754538-f8b9-4891-aa4f-b910ddd9d35e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fq9cv\" (UID: \"0b754538-f8b9-4891-aa4f-b910ddd9d35e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fq9cv" Sep 10 15:50:12.673098 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:12.673033 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b754538-f8b9-4891-aa4f-b910ddd9d35e-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fq9cv\" (UID: \"0b754538-f8b9-4891-aa4f-b910ddd9d35e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fq9cv" Sep 10 15:50:12.673193 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:12.673123 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48cwk\" (UniqueName: \"kubernetes.io/projected/0b754538-f8b9-4891-aa4f-b910ddd9d35e-kube-api-access-48cwk\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fq9cv\" (UID: \"0b754538-f8b9-4891-aa4f-b910ddd9d35e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fq9cv" Sep 10 15:50:12.738833 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:12.738802 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fq9cv" Sep 10 15:50:12.853231 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:12.853192 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:50:13.011209 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:13.011183 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fq9cv"] Sep 10 15:50:13.012954 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:50:13.012932 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b754538_f8b9_4891_aa4f_b910ddd9d35e.slice/crio-94118e67f64ee8291ffea7246b3ebfc1c3d9b7c1875ac2ef0f299bd438b6bce9 WatchSource:0}: Error finding container 94118e67f64ee8291ffea7246b3ebfc1c3d9b7c1875ac2ef0f299bd438b6bce9: Status 404 returned error can't find the container with id 94118e67f64ee8291ffea7246b3ebfc1c3d9b7c1875ac2ef0f299bd438b6bce9 Sep 10 15:50:13.015479 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:13.015439 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 15:50:13.015546 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:13.015528 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 15:50:13.015586 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:13.015574 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 15:50:13.410415 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:13.410381 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fq9cv" event={"ID":"0b754538-f8b9-4891-aa4f-b910ddd9d35e","Type":"ContainerStarted","Data":"cce46f42ce6e7d9125cbeeacfb024d446702e3199bbe923e456dbe31c92b5710"} Sep 10 15:50:13.410604 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:13.410431 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fq9cv" event={"ID":"0b754538-f8b9-4891-aa4f-b910ddd9d35e","Type":"ContainerStarted","Data":"94118e67f64ee8291ffea7246b3ebfc1c3d9b7c1875ac2ef0f299bd438b6bce9"} Sep 10 15:50:13.429307 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:13.429251 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fq9cv" podStartSLOduration=1.429234014 podStartE2EDuration="1.429234014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:50:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:50:13.429006555 +0000 UTC m=+490.782525735" watchObservedRunningTime="2026-09-10 15:50:13.429234014 +0000 UTC m=+490.782753175" Sep 10 15:50:13.739419 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:13.739332 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fq9cv" Sep 10 15:50:14.384755 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:14.384714 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:50:14.517004 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:14.516968 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:50:14.611716 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:14.611681 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:50:14.744494 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:14.744419 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fq9cv" Sep 10 15:50:14.840389 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:14.840350 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 15:50:14.843718 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:14.843689 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-89xpd" Sep 10 15:50:14.845789 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:14.845751 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-pj7x6\"" Sep 10 15:50:14.845919 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:14.845902 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 10 15:50:14.854640 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:14.854618 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 15:50:14.880416 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:14.880389 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 15:50:14.985268 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:14.985234 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/68c81a2d-967e-4f60-a6f7-8ecec49dcecc-config-file\") pod \"limitador-limitador-84994f7ffc-89xpd\" (UID: \"68c81a2d-967e-4f60-a6f7-8ecec49dcecc\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-89xpd" Sep 10 15:50:14.985418 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:14.985301 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dskh7\" (UniqueName: \"kubernetes.io/projected/68c81a2d-967e-4f60-a6f7-8ecec49dcecc-kube-api-access-dskh7\") pod \"limitador-limitador-84994f7ffc-89xpd\" (UID: \"68c81a2d-967e-4f60-a6f7-8ecec49dcecc\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-89xpd" Sep 10 15:50:15.085798 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:15.085759 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/68c81a2d-967e-4f60-a6f7-8ecec49dcecc-config-file\") pod \"limitador-limitador-84994f7ffc-89xpd\" (UID: \"68c81a2d-967e-4f60-a6f7-8ecec49dcecc\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-89xpd" Sep 10 15:50:15.085943 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:15.085832 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dskh7\" (UniqueName: \"kubernetes.io/projected/68c81a2d-967e-4f60-a6f7-8ecec49dcecc-kube-api-access-dskh7\") pod \"limitador-limitador-84994f7ffc-89xpd\" (UID: \"68c81a2d-967e-4f60-a6f7-8ecec49dcecc\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-89xpd" Sep 10 15:50:15.086343 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:15.086324 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/68c81a2d-967e-4f60-a6f7-8ecec49dcecc-config-file\") pod \"limitador-limitador-84994f7ffc-89xpd\" (UID: \"68c81a2d-967e-4f60-a6f7-8ecec49dcecc\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-89xpd" Sep 10 15:50:15.095359 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:15.095338 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dskh7\" (UniqueName: \"kubernetes.io/projected/68c81a2d-967e-4f60-a6f7-8ecec49dcecc-kube-api-access-dskh7\") pod \"limitador-limitador-84994f7ffc-89xpd\" (UID: \"68c81a2d-967e-4f60-a6f7-8ecec49dcecc\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-89xpd" Sep 10 15:50:15.154926 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:15.154898 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-89xpd" Sep 10 15:50:15.261777 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:15.261748 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:50:15.345790 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:15.345695 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:50:15.418699 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:15.418673 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fq9cv" Sep 10 15:50:15.419608 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:15.419593 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fq9cv" Sep 10 15:50:15.428406 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:15.428390 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 15:50:15.430308 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:50:15.430288 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68c81a2d_967e_4f60_a6f7_8ecec49dcecc.slice/crio-0a755a840174507d10c8a34b05d41271c70f26e4c4b3d4fee127b4cf0f679582 WatchSource:0}: Error finding container 0a755a840174507d10c8a34b05d41271c70f26e4c4b3d4fee127b4cf0f679582: Status 404 returned error can't find the container with id 0a755a840174507d10c8a34b05d41271c70f26e4c4b3d4fee127b4cf0f679582 Sep 10 15:50:16.424292 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:16.424259 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-89xpd" event={"ID":"68c81a2d-967e-4f60-a6f7-8ecec49dcecc","Type":"ContainerStarted","Data":"20169b0a55a2f4714dc313d17d9a968c0850e5482cf7bc7520eb5a53aca1030a"} Sep 10 15:50:16.424782 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:16.424308 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-89xpd" event={"ID":"68c81a2d-967e-4f60-a6f7-8ecec49dcecc","Type":"ContainerStarted","Data":"0a755a840174507d10c8a34b05d41271c70f26e4c4b3d4fee127b4cf0f679582"} Sep 10 15:50:16.424782 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:16.424365 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-89xpd" Sep 10 15:50:16.441007 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:16.440965 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-89xpd" podStartSLOduration=1.626485277 podStartE2EDuration="2.440952052s" totalImagesPullingTime="814.466775ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:50:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:50:16.440186518 +0000 UTC m=+493.793705717" watchObservedRunningTime="2026-09-10 15:50:16.440952052 +0000 UTC m=+493.794471213" Sep 10 15:50:27.430713 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:27.430679 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-89xpd" Sep 10 15:50:30.491463 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:30.491422 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:50:30.600134 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:30.600091 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:50:30.928123 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:30.928092 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:50:43.723979 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:43.723938 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:50:43.824586 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:43.824555 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:50:50.487895 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:50.487862 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:50:50.555983 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:50.555947 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:50:51.038200 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:51.038171 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:50:55.148360 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:55.148272 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:50:55.285316 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:50:55.285274 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:51:26.072507 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:51:26.072460 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:51:26.177540 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:51:26.177503 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:51:27.572455 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:51:27.572427 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:51:28.045254 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:51:28.045226 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:51:28.145675 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:51:28.145645 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:51:28.246337 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:51:28.246297 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:51:28.354785 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:51:28.354693 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:51:28.389227 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:51:28.389191 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-9g9rc"] Sep 10 15:51:28.392452 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:51:28.392437 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-9g9rc" Sep 10 15:51:28.394507 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:51:28.394486 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-2x5ws\"" Sep 10 15:51:28.394644 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:51:28.394529 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 10 15:51:28.404204 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:51:28.404182 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-9g9rc"] Sep 10 15:51:28.494174 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:51:28.494143 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5c2d260-e0f6-4e9d-a217-1c4ada9c1ade-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-9g9rc\" (UID: \"b5c2d260-e0f6-4e9d-a217-1c4ada9c1ade\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-9g9rc" Sep 10 15:51:28.494330 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:51:28.494191 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzfls\" (UniqueName: \"kubernetes.io/projected/b5c2d260-e0f6-4e9d-a217-1c4ada9c1ade-kube-api-access-zzfls\") pod \"payload-pre-processing-59fbc47bf8-9g9rc\" (UID: \"b5c2d260-e0f6-4e9d-a217-1c4ada9c1ade\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-9g9rc" Sep 10 15:51:28.595088 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:51:28.595057 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5c2d260-e0f6-4e9d-a217-1c4ada9c1ade-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-9g9rc\" (UID: \"b5c2d260-e0f6-4e9d-a217-1c4ada9c1ade\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-9g9rc" Sep 10 15:51:28.595435 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:51:28.595096 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzfls\" (UniqueName: \"kubernetes.io/projected/b5c2d260-e0f6-4e9d-a217-1c4ada9c1ade-kube-api-access-zzfls\") pod \"payload-pre-processing-59fbc47bf8-9g9rc\" (UID: \"b5c2d260-e0f6-4e9d-a217-1c4ada9c1ade\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-9g9rc" Sep 10 15:51:28.595725 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:51:28.595705 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5c2d260-e0f6-4e9d-a217-1c4ada9c1ade-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-9g9rc\" (UID: \"b5c2d260-e0f6-4e9d-a217-1c4ada9c1ade\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-9g9rc" Sep 10 15:51:28.603510 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:51:28.603484 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzfls\" (UniqueName: \"kubernetes.io/projected/b5c2d260-e0f6-4e9d-a217-1c4ada9c1ade-kube-api-access-zzfls\") pod \"payload-pre-processing-59fbc47bf8-9g9rc\" (UID: \"b5c2d260-e0f6-4e9d-a217-1c4ada9c1ade\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-9g9rc" Sep 10 15:51:28.702668 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:51:28.702612 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-9g9rc" Sep 10 15:51:28.979198 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:51:28.977787 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-9g9rc"] Sep 10 15:51:29.178316 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:51:29.178287 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 15:51:29.756946 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:51:29.756873 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-9g9rc" event={"ID":"b5c2d260-e0f6-4e9d-a217-1c4ada9c1ade","Type":"ContainerStarted","Data":"7947802782aafd80d20d5a1222b1c576581f8d0622f1c4e5f91c1f9ae774f81c"} Sep 10 15:51:31.773936 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:51:31.773900 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-9g9rc" event={"ID":"b5c2d260-e0f6-4e9d-a217-1c4ada9c1ade","Type":"ContainerStarted","Data":"360892b96cb0829ad840c7141f0a146a8e8cc4fe2d00b956cdaefa1bc6fb7253"} Sep 10 15:51:31.774367 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:51:31.773981 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-9g9rc" Sep 10 15:51:31.794069 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:51:31.794024 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-9g9rc" podStartSLOduration=1.960637764 podStartE2EDuration="3.794009946s" totalImagesPullingTime="1.833372182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:51:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:51:31.792514099 +0000 UTC m=+569.146033278" watchObservedRunningTime="2026-09-10 15:51:31.794009946 +0000 UTC m=+569.147529100" Sep 10 15:51:36.720840 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:51:36.720806 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:51:36.781312 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:51:36.781275 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:51:37.805405 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:51:37.805373 2576 generic.go:356] "Generic (PLEG): container finished" podID="0b754538-f8b9-4891-aa4f-b910ddd9d35e" containerID="cce46f42ce6e7d9125cbeeacfb024d446702e3199bbe923e456dbe31c92b5710" exitCode=0 Sep 10 15:51:37.805825 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:51:37.805430 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fq9cv" event={"ID":"0b754538-f8b9-4891-aa4f-b910ddd9d35e","Type":"ContainerDied","Data":"cce46f42ce6e7d9125cbeeacfb024d446702e3199bbe923e456dbe31c92b5710"} Sep 10 15:51:37.805825 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:51:37.805802 2576 scope.go:117] "RemoveContainer" containerID="cce46f42ce6e7d9125cbeeacfb024d446702e3199bbe923e456dbe31c92b5710" Sep 10 15:51:37.806791 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:51:37.806759 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 15:51:37.806902 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:51:37.806831 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 15:51:37.806902 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:51:37.806865 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 15:51:38.630891 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:51:38.630851 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:51:38.730676 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:51:38.730638 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:51:38.814681 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:51:38.814637 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fq9cv" event={"ID":"0b754538-f8b9-4891-aa4f-b910ddd9d35e","Type":"ContainerStarted","Data":"342adcba263c560fa00e959650052dc9f9d450aee350892605c0992882854db6"} Sep 10 15:51:39.739608 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:51:39.739571 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fq9cv" Sep 10 15:51:39.739793 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:51:39.739772 2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-fq9cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Sep 10 15:51:39.739856 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:51:39.739819 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fq9cv" podUID="0b754538-f8b9-4891-aa4f-b910ddd9d35e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Sep 10 15:51:40.739895 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:51:40.739860 2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-fq9cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Sep 10 15:51:40.740260 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:51:40.739923 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fq9cv" podUID="0b754538-f8b9-4891-aa4f-b910ddd9d35e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Sep 10 15:51:41.739646 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:51:41.739607 2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-fq9cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Sep 10 15:51:41.739844 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:51:41.739667 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fq9cv" podUID="0b754538-f8b9-4891-aa4f-b910ddd9d35e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Sep 10 15:51:42.739560 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:51:42.739522 2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-fq9cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Sep 10 15:51:42.739978 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:51:42.739577 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fq9cv" podUID="0b754538-f8b9-4891-aa4f-b910ddd9d35e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Sep 10 15:51:42.780985 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:51:42.780957 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-9g9rc" Sep 10 15:51:43.743843 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:51:43.743815 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fq9cv" Sep 10 15:51:43.744285 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:51:43.744082 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fq9cv" Sep 10 15:51:43.744710 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:51:43.744696 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fq9cv" Sep 10 15:51:49.695248 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:51:49.695206 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:51:50.171278 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:51:50.171242 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 15:52:01.762216 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:52:01.762174 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:52:02.479471 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:52:02.479435 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 15:52:16.291038 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:52:16.290999 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:52:16.970221 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:52:16.970181 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 15:52:21.055750 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:52:21.055694 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:52:22.479046 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:52:22.479012 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 15:52:28.681332 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:52:28.681256 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:52:30.366830 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:52:30.366803 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 15:52:37.464688 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:52:37.464655 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-r6klk"] Sep 10 15:52:37.468328 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:52:37.468309 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6klk" Sep 10 15:52:37.470877 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:52:37.470858 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 10 15:52:37.471311 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:52:37.471293 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-4b9fq\"" Sep 10 15:52:37.471717 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:52:37.471700 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 10 15:52:37.472103 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:52:37.472076 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Sep 10 15:52:37.477965 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:52:37.477945 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-r6klk"] Sep 10 15:52:37.506281 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:52:37.506250 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:52:37.506632 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:52:37.506615 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc071832-44c6-42a3-ba2d-d9a9f98afbc8-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r6klk\" (UID: \"cc071832-44c6-42a3-ba2d-d9a9f98afbc8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6klk" Sep 10 15:52:37.506699 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:52:37.506648 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztr4s\" (UniqueName: \"kubernetes.io/projected/cc071832-44c6-42a3-ba2d-d9a9f98afbc8-kube-api-access-ztr4s\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r6klk\" (UID: \"cc071832-44c6-42a3-ba2d-d9a9f98afbc8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6klk" Sep 10 15:52:37.506699 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:52:37.506674 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc071832-44c6-42a3-ba2d-d9a9f98afbc8-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r6klk\" (UID: \"cc071832-44c6-42a3-ba2d-d9a9f98afbc8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6klk" Sep 10 15:52:37.506815 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:52:37.506797 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc071832-44c6-42a3-ba2d-d9a9f98afbc8-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r6klk\" (UID: \"cc071832-44c6-42a3-ba2d-d9a9f98afbc8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6klk" Sep 10 15:52:37.506855 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:52:37.506825 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc071832-44c6-42a3-ba2d-d9a9f98afbc8-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r6klk\" (UID: \"cc071832-44c6-42a3-ba2d-d9a9f98afbc8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6klk" Sep 10 15:52:37.506855 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:52:37.506842 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc071832-44c6-42a3-ba2d-d9a9f98afbc8-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r6klk\" (UID: \"cc071832-44c6-42a3-ba2d-d9a9f98afbc8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6klk" Sep 10 15:52:37.607622 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:52:37.607594 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc071832-44c6-42a3-ba2d-d9a9f98afbc8-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r6klk\" (UID: \"cc071832-44c6-42a3-ba2d-d9a9f98afbc8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6klk" Sep 10 15:52:37.607767 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:52:37.607653 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc071832-44c6-42a3-ba2d-d9a9f98afbc8-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r6klk\" (UID: \"cc071832-44c6-42a3-ba2d-d9a9f98afbc8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6klk" Sep 10 15:52:37.607767 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:52:37.607674 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc071832-44c6-42a3-ba2d-d9a9f98afbc8-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r6klk\" (UID: \"cc071832-44c6-42a3-ba2d-d9a9f98afbc8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6klk" Sep 10 15:52:37.607767 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:52:37.607689 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc071832-44c6-42a3-ba2d-d9a9f98afbc8-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r6klk\" (UID: \"cc071832-44c6-42a3-ba2d-d9a9f98afbc8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6klk" Sep 10 15:52:37.607767 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:52:37.607710 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc071832-44c6-42a3-ba2d-d9a9f98afbc8-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r6klk\" (UID: \"cc071832-44c6-42a3-ba2d-d9a9f98afbc8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6klk" Sep 10 15:52:37.607767 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:52:37.607762 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztr4s\" (UniqueName: \"kubernetes.io/projected/cc071832-44c6-42a3-ba2d-d9a9f98afbc8-kube-api-access-ztr4s\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r6klk\" (UID: \"cc071832-44c6-42a3-ba2d-d9a9f98afbc8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6klk" Sep 10 15:52:37.608056 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:52:37.608026 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc071832-44c6-42a3-ba2d-d9a9f98afbc8-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r6klk\" (UID: \"cc071832-44c6-42a3-ba2d-d9a9f98afbc8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6klk" Sep 10 15:52:37.608117 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:52:37.608102 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc071832-44c6-42a3-ba2d-d9a9f98afbc8-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r6klk\" (UID: \"cc071832-44c6-42a3-ba2d-d9a9f98afbc8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6klk" Sep 10 15:52:37.608203 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:52:37.608123 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc071832-44c6-42a3-ba2d-d9a9f98afbc8-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r6klk\" (UID: \"cc071832-44c6-42a3-ba2d-d9a9f98afbc8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6klk" Sep 10 15:52:37.610023 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:52:37.610001 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc071832-44c6-42a3-ba2d-d9a9f98afbc8-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r6klk\" (UID: \"cc071832-44c6-42a3-ba2d-d9a9f98afbc8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6klk" Sep 10 15:52:37.610179 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:52:37.610162 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc071832-44c6-42a3-ba2d-d9a9f98afbc8-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r6klk\" (UID: \"cc071832-44c6-42a3-ba2d-d9a9f98afbc8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6klk" Sep 10 15:52:37.617041 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:52:37.617016 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztr4s\" (UniqueName: \"kubernetes.io/projected/cc071832-44c6-42a3-ba2d-d9a9f98afbc8-kube-api-access-ztr4s\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r6klk\" (UID: \"cc071832-44c6-42a3-ba2d-d9a9f98afbc8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6klk" Sep 10 15:52:37.780954 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:52:37.780881 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6klk" Sep 10 15:52:38.054428 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:52:38.054406 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-r6klk"] Sep 10 15:52:38.056498 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:52:38.056466 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc071832_44c6_42a3_ba2d_d9a9f98afbc8.slice/crio-666d28c0712b96519a66a7372bbca5b4d3e3d691374ce12536e9489e8e558fc4 WatchSource:0}: Error finding container 666d28c0712b96519a66a7372bbca5b4d3e3d691374ce12536e9489e8e558fc4: Status 404 returned error can't find the container with id 666d28c0712b96519a66a7372bbca5b4d3e3d691374ce12536e9489e8e558fc4 Sep 10 15:52:38.057908 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:52:38.057890 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 15:52:38.112056 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:52:38.112025 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6klk" event={"ID":"cc071832-44c6-42a3-ba2d-d9a9f98afbc8","Type":"ContainerStarted","Data":"666d28c0712b96519a66a7372bbca5b4d3e3d691374ce12536e9489e8e558fc4"} Sep 10 15:52:39.118433 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:52:39.118403 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6klk" event={"ID":"cc071832-44c6-42a3-ba2d-d9a9f98afbc8","Type":"ContainerStarted","Data":"a1065b3217d34c6c2a9d63b534a0263258a751805ad6e4efd068efb8a7c92a62"} Sep 10 15:52:39.139530 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:52:39.139476 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6klk" podStartSLOduration=1.19952063 podStartE2EDuration="2.139461809s" totalImagesPullingTime="939.941179ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:52:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:52:39.137496771 +0000 UTC m=+636.491015974" watchObservedRunningTime="2026-09-10 15:52:39.139461809 +0000 UTC m=+636.492981004" Sep 10 15:52:39.414953 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:52:39.414920 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 15:52:47.781069 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:52:47.781014 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6klk" Sep 10 15:52:47.781475 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:52:47.781134 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6klk" Sep 10 15:52:47.793535 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:52:47.793508 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6klk" Sep 10 15:52:48.171419 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:52:48.171389 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6klk" Sep 10 15:53:22.153102 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:53:22.153065 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:53:24.676666 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:53:24.676636 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 15:53:37.970451 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:53:37.970415 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:53:41.278386 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:53:41.278351 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:53:41.379643 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:53:41.379612 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:53:44.750590 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:53:44.750558 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:53:44.851247 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:53:44.851220 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:13.367917 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:54:13.367879 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:25.591385 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:54:25.591349 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:25.647214 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:54:25.647163 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:46.286006 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:54:46.285967 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:47.546827 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:54:47.546788 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:03.229285 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:03.229251 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:03.341602 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:03.341572 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:03.569546 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:03.569512 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-789be7-openshift-default-6669946986-bchwf"] Sep 10 15:55:03.572957 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:03.572937 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-789be7-openshift-default-6669946986-bchwf" Sep 10 15:55:03.575249 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:03.575232 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-789be7-openshift-default-dockercfg-rzjwg\"" Sep 10 15:55:03.575638 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:03.575614 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:03.585925 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:03.585904 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-789be7-openshift-default-6669946986-bchwf"] Sep 10 15:55:03.635906 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:03.635872 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d22ca135-fb1d-41c7-bba6-9109e30cfe61-workload-certs\") pod \"e2e-inf-a-789be7-openshift-default-6669946986-bchwf\" (UID: \"d22ca135-fb1d-41c7-bba6-9109e30cfe61\") " pod="openshift-ingress/e2e-inf-a-789be7-openshift-default-6669946986-bchwf" Sep 10 15:55:03.635906 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:03.635906 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d22ca135-fb1d-41c7-bba6-9109e30cfe61-istio-podinfo\") pod \"e2e-inf-a-789be7-openshift-default-6669946986-bchwf\" (UID: \"d22ca135-fb1d-41c7-bba6-9109e30cfe61\") " pod="openshift-ingress/e2e-inf-a-789be7-openshift-default-6669946986-bchwf" Sep 10 15:55:03.636090 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:03.635944 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d22ca135-fb1d-41c7-bba6-9109e30cfe61-istio-data\") pod \"e2e-inf-a-789be7-openshift-default-6669946986-bchwf\" (UID: \"d22ca135-fb1d-41c7-bba6-9109e30cfe61\") " pod="openshift-ingress/e2e-inf-a-789be7-openshift-default-6669946986-bchwf" Sep 10 15:55:03.636090 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:03.636024 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d22ca135-fb1d-41c7-bba6-9109e30cfe61-workload-socket\") pod \"e2e-inf-a-789be7-openshift-default-6669946986-bchwf\" (UID: \"d22ca135-fb1d-41c7-bba6-9109e30cfe61\") " pod="openshift-ingress/e2e-inf-a-789be7-openshift-default-6669946986-bchwf" Sep 10 15:55:03.636090 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:03.636044 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d22ca135-fb1d-41c7-bba6-9109e30cfe61-istiod-ca-cert\") pod \"e2e-inf-a-789be7-openshift-default-6669946986-bchwf\" (UID: \"d22ca135-fb1d-41c7-bba6-9109e30cfe61\") " pod="openshift-ingress/e2e-inf-a-789be7-openshift-default-6669946986-bchwf" Sep 10 15:55:03.636090 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:03.636071 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d22ca135-fb1d-41c7-bba6-9109e30cfe61-istio-token\") pod \"e2e-inf-a-789be7-openshift-default-6669946986-bchwf\" (UID: \"d22ca135-fb1d-41c7-bba6-9109e30cfe61\") " pod="openshift-ingress/e2e-inf-a-789be7-openshift-default-6669946986-bchwf" Sep 10 15:55:03.636262 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:03.636115 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d22ca135-fb1d-41c7-bba6-9109e30cfe61-credential-socket\") pod \"e2e-inf-a-789be7-openshift-default-6669946986-bchwf\" (UID: \"d22ca135-fb1d-41c7-bba6-9109e30cfe61\") " pod="openshift-ingress/e2e-inf-a-789be7-openshift-default-6669946986-bchwf" Sep 10 15:55:03.636262 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:03.636155 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d22ca135-fb1d-41c7-bba6-9109e30cfe61-istio-envoy\") pod \"e2e-inf-a-789be7-openshift-default-6669946986-bchwf\" (UID: \"d22ca135-fb1d-41c7-bba6-9109e30cfe61\") " pod="openshift-ingress/e2e-inf-a-789be7-openshift-default-6669946986-bchwf" Sep 10 15:55:03.636262 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:03.636175 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7qnt\" (UniqueName: \"kubernetes.io/projected/d22ca135-fb1d-41c7-bba6-9109e30cfe61-kube-api-access-v7qnt\") pod \"e2e-inf-a-789be7-openshift-default-6669946986-bchwf\" (UID: \"d22ca135-fb1d-41c7-bba6-9109e30cfe61\") " pod="openshift-ingress/e2e-inf-a-789be7-openshift-default-6669946986-bchwf" Sep 10 15:55:03.644893 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:03.644868 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:03.737627 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:03.737581 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d22ca135-fb1d-41c7-bba6-9109e30cfe61-workload-socket\") pod \"e2e-inf-a-789be7-openshift-default-6669946986-bchwf\" (UID: \"d22ca135-fb1d-41c7-bba6-9109e30cfe61\") " pod="openshift-ingress/e2e-inf-a-789be7-openshift-default-6669946986-bchwf" Sep 10 15:55:03.737627 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:03.737626 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d22ca135-fb1d-41c7-bba6-9109e30cfe61-istiod-ca-cert\") pod \"e2e-inf-a-789be7-openshift-default-6669946986-bchwf\" (UID: \"d22ca135-fb1d-41c7-bba6-9109e30cfe61\") " pod="openshift-ingress/e2e-inf-a-789be7-openshift-default-6669946986-bchwf" Sep 10 15:55:03.737915 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:03.737646 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d22ca135-fb1d-41c7-bba6-9109e30cfe61-istio-token\") pod \"e2e-inf-a-789be7-openshift-default-6669946986-bchwf\" (UID: \"d22ca135-fb1d-41c7-bba6-9109e30cfe61\") " pod="openshift-ingress/e2e-inf-a-789be7-openshift-default-6669946986-bchwf" Sep 10 15:55:03.737915 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:03.737667 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d22ca135-fb1d-41c7-bba6-9109e30cfe61-credential-socket\") pod \"e2e-inf-a-789be7-openshift-default-6669946986-bchwf\" (UID: \"d22ca135-fb1d-41c7-bba6-9109e30cfe61\") " pod="openshift-ingress/e2e-inf-a-789be7-openshift-default-6669946986-bchwf" Sep 10 15:55:03.737915 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:03.737690 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d22ca135-fb1d-41c7-bba6-9109e30cfe61-istio-envoy\") pod \"e2e-inf-a-789be7-openshift-default-6669946986-bchwf\" (UID: \"d22ca135-fb1d-41c7-bba6-9109e30cfe61\") " pod="openshift-ingress/e2e-inf-a-789be7-openshift-default-6669946986-bchwf" Sep 10 15:55:03.737915 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:03.737707 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7qnt\" (UniqueName: \"kubernetes.io/projected/d22ca135-fb1d-41c7-bba6-9109e30cfe61-kube-api-access-v7qnt\") pod \"e2e-inf-a-789be7-openshift-default-6669946986-bchwf\" (UID: \"d22ca135-fb1d-41c7-bba6-9109e30cfe61\") " pod="openshift-ingress/e2e-inf-a-789be7-openshift-default-6669946986-bchwf" Sep 10 15:55:03.737915 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:03.737762 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d22ca135-fb1d-41c7-bba6-9109e30cfe61-workload-certs\") pod \"e2e-inf-a-789be7-openshift-default-6669946986-bchwf\" (UID: \"d22ca135-fb1d-41c7-bba6-9109e30cfe61\") " pod="openshift-ingress/e2e-inf-a-789be7-openshift-default-6669946986-bchwf" Sep 10 15:55:03.737915 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:03.737790 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d22ca135-fb1d-41c7-bba6-9109e30cfe61-istio-podinfo\") pod \"e2e-inf-a-789be7-openshift-default-6669946986-bchwf\" (UID: \"d22ca135-fb1d-41c7-bba6-9109e30cfe61\") " pod="openshift-ingress/e2e-inf-a-789be7-openshift-default-6669946986-bchwf" Sep 10 15:55:03.737915 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:03.737824 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d22ca135-fb1d-41c7-bba6-9109e30cfe61-istio-data\") pod \"e2e-inf-a-789be7-openshift-default-6669946986-bchwf\" (UID: \"d22ca135-fb1d-41c7-bba6-9109e30cfe61\") " pod="openshift-ingress/e2e-inf-a-789be7-openshift-default-6669946986-bchwf" Sep 10 15:55:03.738270 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:03.738084 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d22ca135-fb1d-41c7-bba6-9109e30cfe61-credential-socket\") pod \"e2e-inf-a-789be7-openshift-default-6669946986-bchwf\" (UID: \"d22ca135-fb1d-41c7-bba6-9109e30cfe61\") " pod="openshift-ingress/e2e-inf-a-789be7-openshift-default-6669946986-bchwf" Sep 10 15:55:03.738270 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:03.738137 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d22ca135-fb1d-41c7-bba6-9109e30cfe61-workload-certs\") pod \"e2e-inf-a-789be7-openshift-default-6669946986-bchwf\" (UID: \"d22ca135-fb1d-41c7-bba6-9109e30cfe61\") " pod="openshift-ingress/e2e-inf-a-789be7-openshift-default-6669946986-bchwf" Sep 10 15:55:03.738270 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:03.738199 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d22ca135-fb1d-41c7-bba6-9109e30cfe61-workload-socket\") pod \"e2e-inf-a-789be7-openshift-default-6669946986-bchwf\" (UID: \"d22ca135-fb1d-41c7-bba6-9109e30cfe61\") " pod="openshift-ingress/e2e-inf-a-789be7-openshift-default-6669946986-bchwf" Sep 10 15:55:03.738455 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:03.738438 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d22ca135-fb1d-41c7-bba6-9109e30cfe61-istio-data\") pod \"e2e-inf-a-789be7-openshift-default-6669946986-bchwf\" (UID: \"d22ca135-fb1d-41c7-bba6-9109e30cfe61\") " pod="openshift-ingress/e2e-inf-a-789be7-openshift-default-6669946986-bchwf" Sep 10 15:55:03.738527 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:03.738453 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d22ca135-fb1d-41c7-bba6-9109e30cfe61-istiod-ca-cert\") pod \"e2e-inf-a-789be7-openshift-default-6669946986-bchwf\" (UID: \"d22ca135-fb1d-41c7-bba6-9109e30cfe61\") " pod="openshift-ingress/e2e-inf-a-789be7-openshift-default-6669946986-bchwf" Sep 10 15:55:03.740341 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:03.740320 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d22ca135-fb1d-41c7-bba6-9109e30cfe61-istio-envoy\") pod \"e2e-inf-a-789be7-openshift-default-6669946986-bchwf\" (UID: \"d22ca135-fb1d-41c7-bba6-9109e30cfe61\") " pod="openshift-ingress/e2e-inf-a-789be7-openshift-default-6669946986-bchwf" Sep 10 15:55:03.740600 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:03.740580 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d22ca135-fb1d-41c7-bba6-9109e30cfe61-istio-podinfo\") pod \"e2e-inf-a-789be7-openshift-default-6669946986-bchwf\" (UID: \"d22ca135-fb1d-41c7-bba6-9109e30cfe61\") " pod="openshift-ingress/e2e-inf-a-789be7-openshift-default-6669946986-bchwf" Sep 10 15:55:03.752117 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:03.752084 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d22ca135-fb1d-41c7-bba6-9109e30cfe61-istio-token\") pod \"e2e-inf-a-789be7-openshift-default-6669946986-bchwf\" (UID: \"d22ca135-fb1d-41c7-bba6-9109e30cfe61\") " pod="openshift-ingress/e2e-inf-a-789be7-openshift-default-6669946986-bchwf" Sep 10 15:55:03.752201 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:03.752147 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7qnt\" (UniqueName: \"kubernetes.io/projected/d22ca135-fb1d-41c7-bba6-9109e30cfe61-kube-api-access-v7qnt\") pod \"e2e-inf-a-789be7-openshift-default-6669946986-bchwf\" (UID: \"d22ca135-fb1d-41c7-bba6-9109e30cfe61\") " pod="openshift-ingress/e2e-inf-a-789be7-openshift-default-6669946986-bchwf" Sep 10 15:55:03.883356 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:03.883289 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-789be7-openshift-default-6669946986-bchwf" Sep 10 15:55:04.159600 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:04.159577 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-789be7-openshift-default-6669946986-bchwf"] Sep 10 15:55:04.161227 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:55:04.161198 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd22ca135_fb1d_41c7_bba6_9109e30cfe61.slice/crio-f52f4d03491577b4862ce98dcc9131b5d6fd917ad66530bc78285fc2ea5479bf WatchSource:0}: Error finding container f52f4d03491577b4862ce98dcc9131b5d6fd917ad66530bc78285fc2ea5479bf: Status 404 returned error can't find the container with id f52f4d03491577b4862ce98dcc9131b5d6fd917ad66530bc78285fc2ea5479bf Sep 10 15:55:04.163305 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:04.163278 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 15:55:04.163368 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:04.163342 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 15:55:04.163408 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:04.163371 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 15:55:04.674527 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:04.674490 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:04.768261 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:04.768227 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:04.788927 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:04.788891 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-789be7-openshift-default-6669946986-bchwf" event={"ID":"d22ca135-fb1d-41c7-bba6-9109e30cfe61","Type":"ContainerStarted","Data":"eb2701de50319a1d3a406f55e998e80620caec0dd4623b315a2e37cf4731fb4e"} Sep 10 15:55:04.789092 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:04.788949 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-789be7-openshift-default-6669946986-bchwf" event={"ID":"d22ca135-fb1d-41c7-bba6-9109e30cfe61","Type":"ContainerStarted","Data":"f52f4d03491577b4862ce98dcc9131b5d6fd917ad66530bc78285fc2ea5479bf"} Sep 10 15:55:04.811480 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:04.811430 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-789be7-openshift-default-6669946986-bchwf" podStartSLOduration=1.811414798 podStartE2EDuration="1.811414798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:55:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:55:04.808987131 +0000 UTC m=+782.162506304" watchObservedRunningTime="2026-09-10 15:55:04.811414798 +0000 UTC m=+782.164933958" Sep 10 15:55:04.883789 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:04.883760 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-789be7-openshift-default-6669946986-bchwf" Sep 10 15:55:05.198085 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:05.198038 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:05.889040 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:05.889009 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-789be7-openshift-default-6669946986-bchwf" Sep 10 15:55:06.718455 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:06.718413 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:06.798800 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:06.798772 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-789be7-openshift-default-6669946986-bchwf" Sep 10 15:55:06.799616 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:06.799595 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-789be7-openshift-default-6669946986-bchwf" Sep 10 15:55:06.819238 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:06.819215 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:07.177015 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:07.176984 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 15:55:08.114628 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:08.114594 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:08.214243 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:08.214211 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:08.426801 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:08.426684 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:08.620218 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:08.620184 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:08.719779 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:08.719669 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:08.738509 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:08.738478 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-789be7-6cddb9575d-7m8jc"] Sep 10 15:55:08.742483 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:08.742462 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-789be7-6cddb9575d-7m8jc" Sep 10 15:55:08.744748 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:08.744706 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-789be7\"" Sep 10 15:55:08.744885 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:08.744827 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-789be7-dockercfg-vkmk6\"" Sep 10 15:55:08.748758 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:08.748719 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-789be7-99b575b9c-2ww9p"] Sep 10 15:55:08.752694 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:08.752672 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-789be7-99b575b9c-2ww9p" Sep 10 15:55:08.756190 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:08.756167 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-789be7-6cddb9575d-7m8jc"] Sep 10 15:55:08.766784 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:08.766763 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-789be7-99b575b9c-2ww9p"] Sep 10 15:55:08.789411 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:08.789385 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qffzz\" (UniqueName: \"kubernetes.io/projected/902985a8-c8fb-4afe-8a77-efb13f89f692-kube-api-access-qffzz\") pod \"payload-processing-e2e-inf-a-789be7-99b575b9c-2ww9p\" (UID: \"902985a8-c8fb-4afe-8a77-efb13f89f692\") " pod="openshift-ingress/payload-processing-e2e-inf-a-789be7-99b575b9c-2ww9p" Sep 10 15:55:08.789524 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:08.789421 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/902985a8-c8fb-4afe-8a77-efb13f89f692-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-789be7-99b575b9c-2ww9p\" (UID: \"902985a8-c8fb-4afe-8a77-efb13f89f692\") " pod="openshift-ingress/payload-processing-e2e-inf-a-789be7-99b575b9c-2ww9p" Sep 10 15:55:08.789524 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:08.789459 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbvfh\" (UniqueName: \"kubernetes.io/projected/66c15852-b623-45f9-9289-abafb9ef4329-kube-api-access-cbvfh\") pod \"payload-pre-processing-e2e-inf-a-789be7-6cddb9575d-7m8jc\" (UID: \"66c15852-b623-45f9-9289-abafb9ef4329\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-789be7-6cddb9575d-7m8jc" Sep 10 15:55:08.789662 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:08.789636 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66c15852-b623-45f9-9289-abafb9ef4329-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-789be7-6cddb9575d-7m8jc\" (UID: \"66c15852-b623-45f9-9289-abafb9ef4329\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-789be7-6cddb9575d-7m8jc" Sep 10 15:55:08.890292 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:08.890256 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66c15852-b623-45f9-9289-abafb9ef4329-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-789be7-6cddb9575d-7m8jc\" (UID: \"66c15852-b623-45f9-9289-abafb9ef4329\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-789be7-6cddb9575d-7m8jc" Sep 10 15:55:08.890463 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:08.890307 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qffzz\" (UniqueName: \"kubernetes.io/projected/902985a8-c8fb-4afe-8a77-efb13f89f692-kube-api-access-qffzz\") pod \"payload-processing-e2e-inf-a-789be7-99b575b9c-2ww9p\" (UID: \"902985a8-c8fb-4afe-8a77-efb13f89f692\") " pod="openshift-ingress/payload-processing-e2e-inf-a-789be7-99b575b9c-2ww9p" Sep 10 15:55:08.890463 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:08.890363 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/902985a8-c8fb-4afe-8a77-efb13f89f692-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-789be7-99b575b9c-2ww9p\" (UID: \"902985a8-c8fb-4afe-8a77-efb13f89f692\") " pod="openshift-ingress/payload-processing-e2e-inf-a-789be7-99b575b9c-2ww9p" Sep 10 15:55:08.890463 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:08.890432 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbvfh\" (UniqueName: \"kubernetes.io/projected/66c15852-b623-45f9-9289-abafb9ef4329-kube-api-access-cbvfh\") pod \"payload-pre-processing-e2e-inf-a-789be7-6cddb9575d-7m8jc\" (UID: \"66c15852-b623-45f9-9289-abafb9ef4329\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-789be7-6cddb9575d-7m8jc" Sep 10 15:55:08.891144 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:08.891120 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/902985a8-c8fb-4afe-8a77-efb13f89f692-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-789be7-99b575b9c-2ww9p\" (UID: \"902985a8-c8fb-4afe-8a77-efb13f89f692\") " pod="openshift-ingress/payload-processing-e2e-inf-a-789be7-99b575b9c-2ww9p" Sep 10 15:55:08.891144 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:08.891145 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66c15852-b623-45f9-9289-abafb9ef4329-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-789be7-6cddb9575d-7m8jc\" (UID: \"66c15852-b623-45f9-9289-abafb9ef4329\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-789be7-6cddb9575d-7m8jc" Sep 10 15:55:08.898918 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:08.898890 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbvfh\" (UniqueName: \"kubernetes.io/projected/66c15852-b623-45f9-9289-abafb9ef4329-kube-api-access-cbvfh\") pod \"payload-pre-processing-e2e-inf-a-789be7-6cddb9575d-7m8jc\" (UID: \"66c15852-b623-45f9-9289-abafb9ef4329\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-789be7-6cddb9575d-7m8jc" Sep 10 15:55:08.899445 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:08.899425 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qffzz\" (UniqueName: \"kubernetes.io/projected/902985a8-c8fb-4afe-8a77-efb13f89f692-kube-api-access-qffzz\") pod \"payload-processing-e2e-inf-a-789be7-99b575b9c-2ww9p\" (UID: \"902985a8-c8fb-4afe-8a77-efb13f89f692\") " pod="openshift-ingress/payload-processing-e2e-inf-a-789be7-99b575b9c-2ww9p" Sep 10 15:55:09.056624 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:09.056597 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-789be7-6cddb9575d-7m8jc" Sep 10 15:55:09.068852 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:09.068825 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-789be7-99b575b9c-2ww9p" Sep 10 15:55:09.331637 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:09.331612 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-789be7-6cddb9575d-7m8jc"] Sep 10 15:55:09.333817 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:55:09.333790 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66c15852_b623_45f9_9289_abafb9ef4329.slice/crio-d9065f2329c7af978472d9cd6bc2db60f4b6eb4f4f634e650f09b90a467949d5 WatchSource:0}: Error finding container d9065f2329c7af978472d9cd6bc2db60f4b6eb4f4f634e650f09b90a467949d5: Status 404 returned error can't find the container with id d9065f2329c7af978472d9cd6bc2db60f4b6eb4f4f634e650f09b90a467949d5 Sep 10 15:55:09.351550 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:09.351530 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-789be7-99b575b9c-2ww9p"] Sep 10 15:55:09.353563 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:55:09.353537 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod902985a8_c8fb_4afe_8a77_efb13f89f692.slice/crio-53e1efcb25ee25e94604e580b1e96a0fdaec13d63f115b9a3b943c02475b7666 WatchSource:0}: Error finding container 53e1efcb25ee25e94604e580b1e96a0fdaec13d63f115b9a3b943c02475b7666: Status 404 returned error can't find the container with id 53e1efcb25ee25e94604e580b1e96a0fdaec13d63f115b9a3b943c02475b7666 Sep 10 15:55:09.818819 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:09.818783 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-789be7-6cddb9575d-7m8jc" event={"ID":"66c15852-b623-45f9-9289-abafb9ef4329","Type":"ContainerStarted","Data":"5cd8d47f37e0ac014698331593ce12eaa248263cf255206806ab76213da7f96b"} Sep 10 15:55:09.819016 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:09.818850 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-789be7-6cddb9575d-7m8jc" event={"ID":"66c15852-b623-45f9-9289-abafb9ef4329","Type":"ContainerStarted","Data":"d9065f2329c7af978472d9cd6bc2db60f4b6eb4f4f634e650f09b90a467949d5"} Sep 10 15:55:09.819016 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:09.818985 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-789be7-6cddb9575d-7m8jc" Sep 10 15:55:09.820398 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:09.820375 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-789be7-99b575b9c-2ww9p" event={"ID":"902985a8-c8fb-4afe-8a77-efb13f89f692","Type":"ContainerStarted","Data":"508d4707ec418d22999303ff37b42058c873e066cd5fb3825905f79bfc47cc9b"} Sep 10 15:55:09.820538 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:09.820419 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-789be7-99b575b9c-2ww9p" event={"ID":"902985a8-c8fb-4afe-8a77-efb13f89f692","Type":"ContainerStarted","Data":"53e1efcb25ee25e94604e580b1e96a0fdaec13d63f115b9a3b943c02475b7666"} Sep 10 15:55:09.820538 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:09.820458 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-789be7-99b575b9c-2ww9p" Sep 10 15:55:09.836814 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:09.836772 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-789be7-6cddb9575d-7m8jc" podStartSLOduration=1.836760172 podStartE2EDuration="1.836760172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:55:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:55:09.835343456 +0000 UTC m=+787.188862632" watchObservedRunningTime="2026-09-10 15:55:09.836760172 +0000 UTC m=+787.190279332" Sep 10 15:55:09.853957 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:09.853913 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-789be7-99b575b9c-2ww9p" podStartSLOduration=1.853899406 podStartE2EDuration="1.853899406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:55:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:55:09.85206572 +0000 UTC m=+787.205584907" watchObservedRunningTime="2026-09-10 15:55:09.853899406 +0000 UTC m=+787.207418574" Sep 10 15:55:11.733705 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:11.733674 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7"] Sep 10 15:55:11.737448 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:11.737426 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7" Sep 10 15:55:11.739846 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:11.739825 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-941c6070-openshift-default-dockercfg-q9z5s\"" Sep 10 15:55:11.753081 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:11.753056 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7"] Sep 10 15:55:11.822029 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:11.821244 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bb806e3-78ee-40c8-8106-6d08b3063909-workload-socket\") pod \"e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7\" (UID: \"6bb806e3-78ee-40c8-8106-6d08b3063909\") " pod="openshift-ingress/e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7" Sep 10 15:55:11.822029 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:11.821315 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bb806e3-78ee-40c8-8106-6d08b3063909-workload-certs\") pod \"e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7\" (UID: \"6bb806e3-78ee-40c8-8106-6d08b3063909\") " pod="openshift-ingress/e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7" Sep 10 15:55:11.822029 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:11.821355 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bb806e3-78ee-40c8-8106-6d08b3063909-credential-socket\") pod \"e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7\" (UID: \"6bb806e3-78ee-40c8-8106-6d08b3063909\") " pod="openshift-ingress/e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7" Sep 10 15:55:11.822029 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:11.821378 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bb806e3-78ee-40c8-8106-6d08b3063909-istio-podinfo\") pod \"e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7\" (UID: \"6bb806e3-78ee-40c8-8106-6d08b3063909\") " pod="openshift-ingress/e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7" Sep 10 15:55:11.822029 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:11.821412 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssnrv\" (UniqueName: \"kubernetes.io/projected/6bb806e3-78ee-40c8-8106-6d08b3063909-kube-api-access-ssnrv\") pod \"e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7\" (UID: \"6bb806e3-78ee-40c8-8106-6d08b3063909\") " pod="openshift-ingress/e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7" Sep 10 15:55:11.822029 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:11.821452 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bb806e3-78ee-40c8-8106-6d08b3063909-istio-token\") pod \"e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7\" (UID: \"6bb806e3-78ee-40c8-8106-6d08b3063909\") " pod="openshift-ingress/e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7" Sep 10 15:55:11.822029 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:11.821477 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bb806e3-78ee-40c8-8106-6d08b3063909-istio-envoy\") pod \"e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7\" (UID: \"6bb806e3-78ee-40c8-8106-6d08b3063909\") " pod="openshift-ingress/e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7" Sep 10 15:55:11.822029 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:11.821525 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bb806e3-78ee-40c8-8106-6d08b3063909-istiod-ca-cert\") pod \"e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7\" (UID: \"6bb806e3-78ee-40c8-8106-6d08b3063909\") " pod="openshift-ingress/e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7" Sep 10 15:55:11.822029 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:11.821549 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bb806e3-78ee-40c8-8106-6d08b3063909-istio-data\") pod \"e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7\" (UID: \"6bb806e3-78ee-40c8-8106-6d08b3063909\") " pod="openshift-ingress/e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7" Sep 10 15:55:11.922072 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:11.922036 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bb806e3-78ee-40c8-8106-6d08b3063909-istio-token\") pod \"e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7\" (UID: \"6bb806e3-78ee-40c8-8106-6d08b3063909\") " pod="openshift-ingress/e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7" Sep 10 15:55:11.922072 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:11.922071 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bb806e3-78ee-40c8-8106-6d08b3063909-istio-envoy\") pod \"e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7\" (UID: \"6bb806e3-78ee-40c8-8106-6d08b3063909\") " pod="openshift-ingress/e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7" Sep 10 15:55:11.922309 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:11.922213 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bb806e3-78ee-40c8-8106-6d08b3063909-istiod-ca-cert\") pod \"e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7\" (UID: \"6bb806e3-78ee-40c8-8106-6d08b3063909\") " pod="openshift-ingress/e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7" Sep 10 15:55:11.922309 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:11.922256 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bb806e3-78ee-40c8-8106-6d08b3063909-istio-data\") pod \"e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7\" (UID: \"6bb806e3-78ee-40c8-8106-6d08b3063909\") " pod="openshift-ingress/e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7" Sep 10 15:55:11.922416 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:11.922353 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bb806e3-78ee-40c8-8106-6d08b3063909-workload-socket\") pod \"e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7\" (UID: \"6bb806e3-78ee-40c8-8106-6d08b3063909\") " pod="openshift-ingress/e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7" Sep 10 15:55:11.922416 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:11.922410 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bb806e3-78ee-40c8-8106-6d08b3063909-workload-certs\") pod \"e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7\" (UID: \"6bb806e3-78ee-40c8-8106-6d08b3063909\") " pod="openshift-ingress/e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7" Sep 10 15:55:11.922521 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:11.922454 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bb806e3-78ee-40c8-8106-6d08b3063909-credential-socket\") pod \"e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7\" (UID: \"6bb806e3-78ee-40c8-8106-6d08b3063909\") " pod="openshift-ingress/e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7" Sep 10 15:55:11.922521 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:11.922479 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bb806e3-78ee-40c8-8106-6d08b3063909-istio-podinfo\") pod \"e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7\" (UID: \"6bb806e3-78ee-40c8-8106-6d08b3063909\") " pod="openshift-ingress/e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7" Sep 10 15:55:11.922521 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:11.922516 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssnrv\" (UniqueName: \"kubernetes.io/projected/6bb806e3-78ee-40c8-8106-6d08b3063909-kube-api-access-ssnrv\") pod \"e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7\" (UID: \"6bb806e3-78ee-40c8-8106-6d08b3063909\") " pod="openshift-ingress/e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7" Sep 10 15:55:11.922672 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:11.922646 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bb806e3-78ee-40c8-8106-6d08b3063909-istio-data\") pod \"e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7\" (UID: \"6bb806e3-78ee-40c8-8106-6d08b3063909\") " pod="openshift-ingress/e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7" Sep 10 15:55:11.923021 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:11.923002 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bb806e3-78ee-40c8-8106-6d08b3063909-istiod-ca-cert\") pod \"e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7\" (UID: \"6bb806e3-78ee-40c8-8106-6d08b3063909\") " pod="openshift-ingress/e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7" Sep 10 15:55:11.923122 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:11.923050 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bb806e3-78ee-40c8-8106-6d08b3063909-credential-socket\") pod \"e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7\" (UID: \"6bb806e3-78ee-40c8-8106-6d08b3063909\") " pod="openshift-ingress/e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7" Sep 10 15:55:11.923122 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:11.923098 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bb806e3-78ee-40c8-8106-6d08b3063909-workload-socket\") pod \"e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7\" (UID: \"6bb806e3-78ee-40c8-8106-6d08b3063909\") " pod="openshift-ingress/e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7" Sep 10 15:55:11.923122 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:11.923109 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bb806e3-78ee-40c8-8106-6d08b3063909-workload-certs\") pod \"e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7\" (UID: \"6bb806e3-78ee-40c8-8106-6d08b3063909\") " pod="openshift-ingress/e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7" Sep 10 15:55:11.924405 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:11.924380 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bb806e3-78ee-40c8-8106-6d08b3063909-istio-envoy\") pod \"e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7\" (UID: \"6bb806e3-78ee-40c8-8106-6d08b3063909\") " pod="openshift-ingress/e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7" Sep 10 15:55:11.925029 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:11.925001 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bb806e3-78ee-40c8-8106-6d08b3063909-istio-podinfo\") pod \"e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7\" (UID: \"6bb806e3-78ee-40c8-8106-6d08b3063909\") " pod="openshift-ingress/e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7" Sep 10 15:55:11.946846 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:11.946815 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bb806e3-78ee-40c8-8106-6d08b3063909-istio-token\") pod \"e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7\" (UID: \"6bb806e3-78ee-40c8-8106-6d08b3063909\") " pod="openshift-ingress/e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7" Sep 10 15:55:11.946992 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:11.946975 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssnrv\" (UniqueName: \"kubernetes.io/projected/6bb806e3-78ee-40c8-8106-6d08b3063909-kube-api-access-ssnrv\") pod \"e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7\" (UID: \"6bb806e3-78ee-40c8-8106-6d08b3063909\") " pod="openshift-ingress/e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7" Sep 10 15:55:12.048271 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:12.048239 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7" Sep 10 15:55:12.076478 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:12.076445 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:12.181306 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:12.181270 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:12.297125 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:12.297032 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:12.333959 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:12.333934 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7"] Sep 10 15:55:12.335766 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:55:12.335707 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bb806e3_78ee_40c8_8106_6d08b3063909.slice/crio-4f578a9eefb3d1e5cecc7e9a7afa9b150d117f89ea7e8341b715e2162899988e WatchSource:0}: Error finding container 4f578a9eefb3d1e5cecc7e9a7afa9b150d117f89ea7e8341b715e2162899988e: Status 404 returned error can't find the container with id 4f578a9eefb3d1e5cecc7e9a7afa9b150d117f89ea7e8341b715e2162899988e Sep 10 15:55:12.337908 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:12.337874 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 15:55:12.337994 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:12.337939 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 15:55:12.337994 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:12.337968 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 15:55:12.840498 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:12.840461 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7" event={"ID":"6bb806e3-78ee-40c8-8106-6d08b3063909","Type":"ContainerStarted","Data":"8e61408b51cd872b3851de3bce527873fca78a630a92439413798bdcccecf5a0"} Sep 10 15:55:12.840995 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:12.840528 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7" event={"ID":"6bb806e3-78ee-40c8-8106-6d08b3063909","Type":"ContainerStarted","Data":"4f578a9eefb3d1e5cecc7e9a7afa9b150d117f89ea7e8341b715e2162899988e"} Sep 10 15:55:12.863897 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:12.863836 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7" podStartSLOduration=1.863812787 podStartE2EDuration="1.863812787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:55:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:55:12.860569113 +0000 UTC m=+790.214088294" watchObservedRunningTime="2026-09-10 15:55:12.863812787 +0000 UTC m=+790.217331949" Sep 10 15:55:12.955814 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:12.955779 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:13.048472 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:13.048434 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7" Sep 10 15:55:13.053446 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:13.053421 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7" Sep 10 15:55:13.845918 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:13.845889 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7" Sep 10 15:55:13.846995 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:13.846972 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7" Sep 10 15:55:14.235984 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:14.235892 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:14.340424 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:14.340394 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:15.544642 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:15.544611 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:15.648998 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:15.648959 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:15.893265 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:15.893179 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:16.001452 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:16.001412 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:16.112449 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:16.112413 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:16.211785 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:16.211683 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:16.318189 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:16.318150 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:16.320755 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:16.320186 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-941c6070-69589b98c-qxb2b"] Sep 10 15:55:16.325342 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:16.325322 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-941c6070-69589b98c-qxb2b" Sep 10 15:55:16.327838 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:16.327819 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-941c6070-dockercfg-6gxx6\"" Sep 10 15:55:16.328079 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:16.328066 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-941c6070\"" Sep 10 15:55:16.336135 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:16.336114 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-941c6070-69589b98c-qxb2b"] Sep 10 15:55:16.367839 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:16.367806 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d0e8c4b-e3a9-439c-9a19-c1fab80f54d0-plugins-config-volume\") pod \"payload-processing-e2e-ait-941c6070-69589b98c-qxb2b\" (UID: \"1d0e8c4b-e3a9-439c-9a19-c1fab80f54d0\") " pod="openshift-ingress/payload-processing-e2e-ait-941c6070-69589b98c-qxb2b" Sep 10 15:55:16.367960 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:16.367875 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd8xs\" (UniqueName: \"kubernetes.io/projected/1d0e8c4b-e3a9-439c-9a19-c1fab80f54d0-kube-api-access-hd8xs\") pod \"payload-processing-e2e-ait-941c6070-69589b98c-qxb2b\" (UID: \"1d0e8c4b-e3a9-439c-9a19-c1fab80f54d0\") " pod="openshift-ingress/payload-processing-e2e-ait-941c6070-69589b98c-qxb2b" Sep 10 15:55:16.468884 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:16.468790 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d0e8c4b-e3a9-439c-9a19-c1fab80f54d0-plugins-config-volume\") pod \"payload-processing-e2e-ait-941c6070-69589b98c-qxb2b\" (UID: \"1d0e8c4b-e3a9-439c-9a19-c1fab80f54d0\") " pod="openshift-ingress/payload-processing-e2e-ait-941c6070-69589b98c-qxb2b" Sep 10 15:55:16.468884 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:16.468871 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd8xs\" (UniqueName: \"kubernetes.io/projected/1d0e8c4b-e3a9-439c-9a19-c1fab80f54d0-kube-api-access-hd8xs\") pod \"payload-processing-e2e-ait-941c6070-69589b98c-qxb2b\" (UID: \"1d0e8c4b-e3a9-439c-9a19-c1fab80f54d0\") " pod="openshift-ingress/payload-processing-e2e-ait-941c6070-69589b98c-qxb2b" Sep 10 15:55:16.469844 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:16.469818 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d0e8c4b-e3a9-439c-9a19-c1fab80f54d0-plugins-config-volume\") pod \"payload-processing-e2e-ait-941c6070-69589b98c-qxb2b\" (UID: \"1d0e8c4b-e3a9-439c-9a19-c1fab80f54d0\") " pod="openshift-ingress/payload-processing-e2e-ait-941c6070-69589b98c-qxb2b" Sep 10 15:55:16.477330 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:16.477307 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd8xs\" (UniqueName: \"kubernetes.io/projected/1d0e8c4b-e3a9-439c-9a19-c1fab80f54d0-kube-api-access-hd8xs\") pod \"payload-processing-e2e-ait-941c6070-69589b98c-qxb2b\" (UID: \"1d0e8c4b-e3a9-439c-9a19-c1fab80f54d0\") " pod="openshift-ingress/payload-processing-e2e-ait-941c6070-69589b98c-qxb2b" Sep 10 15:55:16.637792 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:16.637760 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-941c6070-69589b98c-qxb2b" Sep 10 15:55:16.924422 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:16.924393 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-941c6070-69589b98c-qxb2b"] Sep 10 15:55:16.926360 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:55:16.926331 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d0e8c4b_e3a9_439c_9a19_c1fab80f54d0.slice/crio-bf21ed434eaacc02f4b8598c02f357b3f338a6cfecbc1576d2e05fca1f5228ee WatchSource:0}: Error finding container bf21ed434eaacc02f4b8598c02f357b3f338a6cfecbc1576d2e05fca1f5228ee: Status 404 returned error can't find the container with id bf21ed434eaacc02f4b8598c02f357b3f338a6cfecbc1576d2e05fca1f5228ee Sep 10 15:55:17.187765 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:17.187674 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 15:55:17.868597 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:17.868556 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-941c6070-69589b98c-qxb2b" event={"ID":"1d0e8c4b-e3a9-439c-9a19-c1fab80f54d0","Type":"ContainerStarted","Data":"574df7ef8bffb42c0245afdb3d20f480bb5db873f0b106740c72d1737352d446"} Sep 10 15:55:17.869006 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:17.868637 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-941c6070-69589b98c-qxb2b" event={"ID":"1d0e8c4b-e3a9-439c-9a19-c1fab80f54d0","Type":"ContainerStarted","Data":"bf21ed434eaacc02f4b8598c02f357b3f338a6cfecbc1576d2e05fca1f5228ee"} Sep 10 15:55:17.869006 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:17.868659 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-941c6070-69589b98c-qxb2b" Sep 10 15:55:17.894243 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:17.894191 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-941c6070-69589b98c-qxb2b" podStartSLOduration=1.894175162 podStartE2EDuration="1.894175162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:55:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:55:17.891175722 +0000 UTC m=+795.244694902" watchObservedRunningTime="2026-09-10 15:55:17.894175162 +0000 UTC m=+795.247694372" Sep 10 15:55:18.184653 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:18.184562 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:18.274160 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:18.274134 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:19.539759 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:19.539698 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:19.639183 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:19.639141 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:19.829990 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:19.829910 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:20.134838 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:20.134762 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:20.269076 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:20.269043 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-314824-55c4c7d84f-szjbr"] Sep 10 15:55:20.273983 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:20.273962 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-314824-55c4c7d84f-szjbr" Sep 10 15:55:20.276289 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:20.276261 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-314824-dockercfg-4krws\"" Sep 10 15:55:20.276400 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:20.276325 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-314824\"" Sep 10 15:55:20.289933 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:20.289903 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-314824-55c4c7d84f-szjbr"] Sep 10 15:55:20.409898 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:20.409822 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thjqg\" (UniqueName: \"kubernetes.io/projected/e5a17db3-3e2c-421a-bacf-7b9844a8a388-kube-api-access-thjqg\") pod \"payload-processing-e2e-autores-314824-55c4c7d84f-szjbr\" (UID: \"e5a17db3-3e2c-421a-bacf-7b9844a8a388\") " pod="openshift-ingress/payload-processing-e2e-autores-314824-55c4c7d84f-szjbr" Sep 10 15:55:20.409898 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:20.409890 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5a17db3-3e2c-421a-bacf-7b9844a8a388-plugins-config-volume\") pod \"payload-processing-e2e-autores-314824-55c4c7d84f-szjbr\" (UID: \"e5a17db3-3e2c-421a-bacf-7b9844a8a388\") " pod="openshift-ingress/payload-processing-e2e-autores-314824-55c4c7d84f-szjbr" Sep 10 15:55:20.511249 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:20.511216 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5a17db3-3e2c-421a-bacf-7b9844a8a388-plugins-config-volume\") pod \"payload-processing-e2e-autores-314824-55c4c7d84f-szjbr\" (UID: \"e5a17db3-3e2c-421a-bacf-7b9844a8a388\") " pod="openshift-ingress/payload-processing-e2e-autores-314824-55c4c7d84f-szjbr" Sep 10 15:55:20.511429 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:20.511302 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thjqg\" (UniqueName: \"kubernetes.io/projected/e5a17db3-3e2c-421a-bacf-7b9844a8a388-kube-api-access-thjqg\") pod \"payload-processing-e2e-autores-314824-55c4c7d84f-szjbr\" (UID: \"e5a17db3-3e2c-421a-bacf-7b9844a8a388\") " pod="openshift-ingress/payload-processing-e2e-autores-314824-55c4c7d84f-szjbr" Sep 10 15:55:20.511965 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:20.511944 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5a17db3-3e2c-421a-bacf-7b9844a8a388-plugins-config-volume\") pod \"payload-processing-e2e-autores-314824-55c4c7d84f-szjbr\" (UID: \"e5a17db3-3e2c-421a-bacf-7b9844a8a388\") " pod="openshift-ingress/payload-processing-e2e-autores-314824-55c4c7d84f-szjbr" Sep 10 15:55:20.520316 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:20.520286 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thjqg\" (UniqueName: \"kubernetes.io/projected/e5a17db3-3e2c-421a-bacf-7b9844a8a388-kube-api-access-thjqg\") pod \"payload-processing-e2e-autores-314824-55c4c7d84f-szjbr\" (UID: \"e5a17db3-3e2c-421a-bacf-7b9844a8a388\") " pod="openshift-ingress/payload-processing-e2e-autores-314824-55c4c7d84f-szjbr" Sep 10 15:55:20.588511 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:20.588479 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-314824-55c4c7d84f-szjbr" Sep 10 15:55:20.726493 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:20.726418 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:20.829317 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:20.829291 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-789be7-6cddb9575d-7m8jc" Sep 10 15:55:20.829490 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:20.829369 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-789be7-99b575b9c-2ww9p" Sep 10 15:55:20.871449 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:20.871357 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-314824-55c4c7d84f-szjbr"] Sep 10 15:55:20.872439 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:55:20.872408 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5a17db3_3e2c_421a_bacf_7b9844a8a388.slice/crio-7dd6b3ed0d30d1739aaa43b0a01ae42a35ee6ad801ae223f751e7f8ad4c7d1eb WatchSource:0}: Error finding container 7dd6b3ed0d30d1739aaa43b0a01ae42a35ee6ad801ae223f751e7f8ad4c7d1eb: Status 404 returned error can't find the container with id 7dd6b3ed0d30d1739aaa43b0a01ae42a35ee6ad801ae223f751e7f8ad4c7d1eb Sep 10 15:55:20.890243 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:20.890178 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-314824-55c4c7d84f-szjbr" event={"ID":"e5a17db3-3e2c-421a-bacf-7b9844a8a388","Type":"ContainerStarted","Data":"7dd6b3ed0d30d1739aaa43b0a01ae42a35ee6ad801ae223f751e7f8ad4c7d1eb"} Sep 10 15:55:21.129811 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:21.129779 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:21.800385 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:21.800345 2576 patch_prober.go:28] interesting pod/e2e-inf-a-789be7-openshift-default-6669946986-bchwf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Sep 10 15:55:21.800828 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:21.800413 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-789be7-openshift-default-6669946986-bchwf" podUID="d22ca135-fb1d-41c7-bba6-9109e30cfe61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Sep 10 15:55:21.898035 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:21.897997 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-314824-55c4c7d84f-szjbr" event={"ID":"e5a17db3-3e2c-421a-bacf-7b9844a8a388","Type":"ContainerStarted","Data":"f9bf73f298e8d429882167b9e56b5b3dfa6937d108e152c2fec14251a597fcd2"} Sep 10 15:55:21.898235 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:21.898115 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-314824-55c4c7d84f-szjbr" Sep 10 15:55:21.899857 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:21.899827 2576 generic.go:356] "Generic (PLEG): container finished" podID="d22ca135-fb1d-41c7-bba6-9109e30cfe61" containerID="eb2701de50319a1d3a406f55e998e80620caec0dd4623b315a2e37cf4731fb4e" exitCode=0 Sep 10 15:55:21.899997 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:21.899870 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-789be7-openshift-default-6669946986-bchwf" event={"ID":"d22ca135-fb1d-41c7-bba6-9109e30cfe61","Type":"ContainerDied","Data":"eb2701de50319a1d3a406f55e998e80620caec0dd4623b315a2e37cf4731fb4e"} Sep 10 15:55:21.900233 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:21.900215 2576 scope.go:117] "RemoveContainer" containerID="eb2701de50319a1d3a406f55e998e80620caec0dd4623b315a2e37cf4731fb4e" Sep 10 15:55:21.901384 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:21.901354 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 15:55:21.901473 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:21.901424 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 15:55:21.901531 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:21.901474 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 15:55:21.921651 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:21.921587 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-314824-55c4c7d84f-szjbr" podStartSLOduration=1.921572348 podStartE2EDuration="1.921572348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:55:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:55:21.919538733 +0000 UTC m=+799.273057912" watchObservedRunningTime="2026-09-10 15:55:21.921572348 +0000 UTC m=+799.275091509" Sep 10 15:55:22.911769 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:22.911705 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-789be7-openshift-default-6669946986-bchwf" event={"ID":"d22ca135-fb1d-41c7-bba6-9109e30cfe61","Type":"ContainerStarted","Data":"d14c027af6ffee6e7bd5c45dca0e656b1bd8a899b84462474dbcf4e580f38213"} Sep 10 15:55:23.394511 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:23.394479 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:23.510476 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:23.510441 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:23.596282 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:23.596251 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:23.729421 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:23.729340 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:23.883709 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:23.883676 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-789be7-openshift-default-6669946986-bchwf" Sep 10 15:55:23.883914 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:23.883892 2576 patch_prober.go:28] interesting pod/e2e-inf-a-789be7-openshift-default-6669946986-bchwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Sep 10 15:55:23.883985 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:23.883941 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-789be7-openshift-default-6669946986-bchwf" podUID="d22ca135-fb1d-41c7-bba6-9109e30cfe61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Sep 10 15:55:23.926586 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:23.926544 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:24.068246 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:24.068209 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:24.171468 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:24.171432 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:24.267442 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:24.267370 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:24.884380 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:24.884348 2576 patch_prober.go:28] interesting pod/e2e-inf-a-789be7-openshift-default-6669946986-bchwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Sep 10 15:55:24.884586 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:24.884408 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-789be7-openshift-default-6669946986-bchwf" podUID="d22ca135-fb1d-41c7-bba6-9109e30cfe61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Sep 10 15:55:25.884669 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:25.884613 2576 patch_prober.go:28] interesting pod/e2e-inf-a-789be7-openshift-default-6669946986-bchwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Sep 10 15:55:25.885102 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:25.884688 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-789be7-openshift-default-6669946986-bchwf" podUID="d22ca135-fb1d-41c7-bba6-9109e30cfe61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Sep 10 15:55:26.883957 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:26.883914 2576 patch_prober.go:28] interesting pod/e2e-inf-a-789be7-openshift-default-6669946986-bchwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Sep 10 15:55:26.884129 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:26.883988 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-789be7-openshift-default-6669946986-bchwf" podUID="d22ca135-fb1d-41c7-bba6-9109e30cfe61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Sep 10 15:55:26.958196 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:26.958166 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:27.883953 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:27.883921 2576 patch_prober.go:28] interesting pod/e2e-inf-a-789be7-openshift-default-6669946986-bchwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Sep 10 15:55:27.884139 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:27.883990 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-789be7-openshift-default-6669946986-bchwf" podUID="d22ca135-fb1d-41c7-bba6-9109e30cfe61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Sep 10 15:55:28.877223 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:28.877195 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-941c6070-69589b98c-qxb2b" Sep 10 15:55:28.887851 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:28.887817 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-789be7-openshift-default-6669946986-bchwf" Sep 10 15:55:28.888103 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:28.888085 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-789be7-openshift-default-6669946986-bchwf" Sep 10 15:55:28.888788 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:28.888774 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-789be7-openshift-default-6669946986-bchwf" Sep 10 15:55:28.932566 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:28.932529 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:29.915113 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:29.915074 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:30.015049 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:30.015012 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:30.176234 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:30.176141 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:31.280571 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:31.280534 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 15:55:31.730178 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:31.730142 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:32.054276 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:32.054229 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:32.331774 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:32.331638 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:32.574388 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:32.574344 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:32.913680 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:32.913607 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-314824-55c4c7d84f-szjbr" Sep 10 15:55:32.981689 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:32.981660 2576 generic.go:356] "Generic (PLEG): container finished" podID="6bb806e3-78ee-40c8-8106-6d08b3063909" containerID="8e61408b51cd872b3851de3bce527873fca78a630a92439413798bdcccecf5a0" exitCode=0 Sep 10 15:55:32.981895 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:32.981759 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7" event={"ID":"6bb806e3-78ee-40c8-8106-6d08b3063909","Type":"ContainerDied","Data":"8e61408b51cd872b3851de3bce527873fca78a630a92439413798bdcccecf5a0"} Sep 10 15:55:32.982193 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:32.982172 2576 scope.go:117] "RemoveContainer" containerID="8e61408b51cd872b3851de3bce527873fca78a630a92439413798bdcccecf5a0" Sep 10 15:55:32.983390 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:32.983357 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 15:55:32.983461 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:32.983422 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 15:55:32.983461 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:32.983451 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 15:55:33.989426 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:33.989391 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7" event={"ID":"6bb806e3-78ee-40c8-8106-6d08b3063909","Type":"ContainerStarted","Data":"29c94d5584368774a803470487b1d2ef7d43b912cd968c5310c4120fd027a10c"} Sep 10 15:55:34.049197 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:34.049163 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7" Sep 10 15:55:34.049393 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:34.049371 2576 patch_prober.go:28] interesting pod/e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Sep 10 15:55:34.049440 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:34.049419 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7" podUID="6bb806e3-78ee-40c8-8106-6d08b3063909" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Sep 10 15:55:35.048581 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:35.048547 2576 patch_prober.go:28] interesting pod/e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Sep 10 15:55:35.048996 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:35.048601 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7" podUID="6bb806e3-78ee-40c8-8106-6d08b3063909" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Sep 10 15:55:35.912916 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:35.912872 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:36.007300 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:36.007252 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:36.049623 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:36.049584 2576 patch_prober.go:28] interesting pod/e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Sep 10 15:55:36.050057 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:36.049641 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7" podUID="6bb806e3-78ee-40c8-8106-6d08b3063909" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Sep 10 15:55:36.789023 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:36.788985 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:37.049065 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:37.048980 2576 patch_prober.go:28] interesting pod/e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Sep 10 15:55:37.049065 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:37.049036 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7" podUID="6bb806e3-78ee-40c8-8106-6d08b3063909" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Sep 10 15:55:37.658491 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:37.658449 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:37.761756 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:37.761696 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:37.944216 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:37.944127 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:38.049041 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:38.049009 2576 patch_prober.go:28] interesting pod/e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Sep 10 15:55:38.049215 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:38.049074 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7" podUID="6bb806e3-78ee-40c8-8106-6d08b3063909" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Sep 10 15:55:38.200449 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:38.200353 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:38.302694 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:38.302656 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:38.328356 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:38.328321 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5d6f34-7ffc476787-xcx97"] Sep 10 15:55:38.332642 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:38.332620 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5d6f34-7ffc476787-xcx97" Sep 10 15:55:38.334924 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:38.334903 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-5d6f34\"" Sep 10 15:55:38.335308 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:38.335292 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-5d6f34-dockercfg-54gqz\"" Sep 10 15:55:38.345957 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:38.345936 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5d6f34-7ffc476787-xcx97"] Sep 10 15:55:38.419795 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:38.417344 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5462ffa6-e222-4771-add5-6afcf521ce8f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5d6f34-7ffc476787-xcx97\" (UID: \"5462ffa6-e222-4771-add5-6afcf521ce8f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5d6f34-7ffc476787-xcx97" Sep 10 15:55:38.419795 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:38.417546 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg4lf\" (UniqueName: \"kubernetes.io/projected/5462ffa6-e222-4771-add5-6afcf521ce8f-kube-api-access-qg4lf\") pod \"payload-processing-e2e-inf-b-5d6f34-7ffc476787-xcx97\" (UID: \"5462ffa6-e222-4771-add5-6afcf521ce8f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5d6f34-7ffc476787-xcx97" Sep 10 15:55:38.520217 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:38.518769 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg4lf\" (UniqueName: \"kubernetes.io/projected/5462ffa6-e222-4771-add5-6afcf521ce8f-kube-api-access-qg4lf\") pod \"payload-processing-e2e-inf-b-5d6f34-7ffc476787-xcx97\" (UID: \"5462ffa6-e222-4771-add5-6afcf521ce8f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5d6f34-7ffc476787-xcx97" Sep 10 15:55:38.520217 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:38.518869 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5462ffa6-e222-4771-add5-6afcf521ce8f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5d6f34-7ffc476787-xcx97\" (UID: \"5462ffa6-e222-4771-add5-6afcf521ce8f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5d6f34-7ffc476787-xcx97" Sep 10 15:55:38.520217 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:38.519839 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5462ffa6-e222-4771-add5-6afcf521ce8f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5d6f34-7ffc476787-xcx97\" (UID: \"5462ffa6-e222-4771-add5-6afcf521ce8f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5d6f34-7ffc476787-xcx97" Sep 10 15:55:38.530482 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:38.530430 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg4lf\" (UniqueName: \"kubernetes.io/projected/5462ffa6-e222-4771-add5-6afcf521ce8f-kube-api-access-qg4lf\") pod \"payload-processing-e2e-inf-b-5d6f34-7ffc476787-xcx97\" (UID: \"5462ffa6-e222-4771-add5-6afcf521ce8f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5d6f34-7ffc476787-xcx97" Sep 10 15:55:38.643963 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:38.643920 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5d6f34-7ffc476787-xcx97" Sep 10 15:55:38.733516 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:38.733296 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:38.916572 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:38.916540 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5d6f34-7ffc476787-xcx97"] Sep 10 15:55:38.919682 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:55:38.919646 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5462ffa6_e222_4771_add5_6afcf521ce8f.slice/crio-abd5310d6d5a623f3cb82ce196eb09940cf31c9f97fc857950f4cc8a63cbf528 WatchSource:0}: Error finding container abd5310d6d5a623f3cb82ce196eb09940cf31c9f97fc857950f4cc8a63cbf528: Status 404 returned error can't find the container with id abd5310d6d5a623f3cb82ce196eb09940cf31c9f97fc857950f4cc8a63cbf528 Sep 10 15:55:39.027126 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:39.027088 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5d6f34-7ffc476787-xcx97" event={"ID":"5462ffa6-e222-4771-add5-6afcf521ce8f","Type":"ContainerStarted","Data":"1da1fc448a1eae5f945d82d98f34e16ac35e96faf5935d7a44e39c16bede035b"} Sep 10 15:55:39.027303 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:39.027155 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5d6f34-7ffc476787-xcx97" event={"ID":"5462ffa6-e222-4771-add5-6afcf521ce8f","Type":"ContainerStarted","Data":"abd5310d6d5a623f3cb82ce196eb09940cf31c9f97fc857950f4cc8a63cbf528"} Sep 10 15:55:39.027303 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:39.027186 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-5d6f34-7ffc476787-xcx97" Sep 10 15:55:39.046089 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:39.045813 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-5d6f34-7ffc476787-xcx97" podStartSLOduration=1.04579457 podStartE2EDuration="1.04579457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:55:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:55:39.042859008 +0000 UTC m=+816.396378196" watchObservedRunningTime="2026-09-10 15:55:39.04579457 +0000 UTC m=+816.399313730" Sep 10 15:55:39.227469 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:39.227386 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7" Sep 10 15:55:39.228098 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:39.228070 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7" Sep 10 15:55:39.229930 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:39.229907 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7" Sep 10 15:55:39.626744 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:39.626694 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:40.406491 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:40.406452 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:41.634754 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:41.634703 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:43.084162 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:43.084127 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:50.035337 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:50.035307 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-5d6f34-7ffc476787-xcx97" Sep 10 15:55:50.657918 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:50.657890 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:52.081657 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:52.081620 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 15:55:55.913211 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:55.913178 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:56.552268 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:56.552233 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-941c6070-69589b98c-qxb2b"] Sep 10 15:55:56.552503 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:56.552479 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-941c6070-69589b98c-qxb2b" podUID="1d0e8c4b-e3a9-439c-9a19-c1fab80f54d0" containerName="payload-processing" containerID="cri-o://574df7ef8bffb42c0245afdb3d20f480bb5db873f0b106740c72d1737352d446" gracePeriod=30 Sep 10 15:55:56.593661 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:56.593629 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:56.694644 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:56.694609 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:56.760945 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:56.760915 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:56.809439 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:56.809368 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-941c6070-69589b98c-qxb2b" Sep 10 15:55:56.854394 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:56.854362 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7"] Sep 10 15:55:56.854883 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:56.854854 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7" podUID="6bb806e3-78ee-40c8-8106-6d08b3063909" containerName="istio-proxy" containerID="cri-o://29c94d5584368774a803470487b1d2ef7d43b912cd968c5310c4120fd027a10c" gracePeriod=30 Sep 10 15:55:56.874646 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:56.874610 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:56.904663 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:56.904631 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hd8xs\" (UniqueName: \"kubernetes.io/projected/1d0e8c4b-e3a9-439c-9a19-c1fab80f54d0-kube-api-access-hd8xs\") pod \"1d0e8c4b-e3a9-439c-9a19-c1fab80f54d0\" (UID: \"1d0e8c4b-e3a9-439c-9a19-c1fab80f54d0\") " Sep 10 15:55:56.904851 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:56.904699 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d0e8c4b-e3a9-439c-9a19-c1fab80f54d0-plugins-config-volume\") pod \"1d0e8c4b-e3a9-439c-9a19-c1fab80f54d0\" (UID: \"1d0e8c4b-e3a9-439c-9a19-c1fab80f54d0\") " Sep 10 15:55:56.905050 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:56.905029 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d0e8c4b-e3a9-439c-9a19-c1fab80f54d0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1d0e8c4b-e3a9-439c-9a19-c1fab80f54d0" (UID: "1d0e8c4b-e3a9-439c-9a19-c1fab80f54d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:55:56.906938 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:56.906915 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d0e8c4b-e3a9-439c-9a19-c1fab80f54d0-kube-api-access-hd8xs" (OuterVolumeSpecName: "kube-api-access-hd8xs") pod "1d0e8c4b-e3a9-439c-9a19-c1fab80f54d0" (UID: "1d0e8c4b-e3a9-439c-9a19-c1fab80f54d0"). InnerVolumeSpecName "kube-api-access-hd8xs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:55:57.006355 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:57.006307 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hd8xs\" (UniqueName: \"kubernetes.io/projected/1d0e8c4b-e3a9-439c-9a19-c1fab80f54d0-kube-api-access-hd8xs\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:55:57.006355 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:57.006344 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d0e8c4b-e3a9-439c-9a19-c1fab80f54d0-plugins-config-volume\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:55:57.124639 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:57.124542 2576 generic.go:356] "Generic (PLEG): container finished" podID="1d0e8c4b-e3a9-439c-9a19-c1fab80f54d0" containerID="574df7ef8bffb42c0245afdb3d20f480bb5db873f0b106740c72d1737352d446" exitCode=0 Sep 10 15:55:57.124639 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:57.124603 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-941c6070-69589b98c-qxb2b" Sep 10 15:55:57.124845 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:57.124636 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-941c6070-69589b98c-qxb2b" event={"ID":"1d0e8c4b-e3a9-439c-9a19-c1fab80f54d0","Type":"ContainerDied","Data":"574df7ef8bffb42c0245afdb3d20f480bb5db873f0b106740c72d1737352d446"} Sep 10 15:55:57.124845 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:57.124720 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-941c6070-69589b98c-qxb2b" event={"ID":"1d0e8c4b-e3a9-439c-9a19-c1fab80f54d0","Type":"ContainerDied","Data":"bf21ed434eaacc02f4b8598c02f357b3f338a6cfecbc1576d2e05fca1f5228ee"} Sep 10 15:55:57.124845 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:57.124763 2576 scope.go:117] "RemoveContainer" containerID="574df7ef8bffb42c0245afdb3d20f480bb5db873f0b106740c72d1737352d446" Sep 10 15:55:57.133937 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:57.133921 2576 scope.go:117] "RemoveContainer" containerID="574df7ef8bffb42c0245afdb3d20f480bb5db873f0b106740c72d1737352d446" Sep 10 15:55:57.134191 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:55:57.134170 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"574df7ef8bffb42c0245afdb3d20f480bb5db873f0b106740c72d1737352d446\": container with ID starting with 574df7ef8bffb42c0245afdb3d20f480bb5db873f0b106740c72d1737352d446 not found: ID does not exist" containerID="574df7ef8bffb42c0245afdb3d20f480bb5db873f0b106740c72d1737352d446" Sep 10 15:55:57.134236 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:57.134201 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"574df7ef8bffb42c0245afdb3d20f480bb5db873f0b106740c72d1737352d446"} err="failed to get container status \"574df7ef8bffb42c0245afdb3d20f480bb5db873f0b106740c72d1737352d446\": rpc error: code = NotFound desc = could not find container \"574df7ef8bffb42c0245afdb3d20f480bb5db873f0b106740c72d1737352d446\": container with ID starting with 574df7ef8bffb42c0245afdb3d20f480bb5db873f0b106740c72d1737352d446 not found: ID does not exist" Sep 10 15:55:57.145361 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:57.145337 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-941c6070-69589b98c-qxb2b"] Sep 10 15:55:57.149404 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:57.149382 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-941c6070-69589b98c-qxb2b"] Sep 10 15:55:57.169672 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:57.169646 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d0e8c4b-e3a9-439c-9a19-c1fab80f54d0" path="/var/lib/kubelet/pods/1d0e8c4b-e3a9-439c-9a19-c1fab80f54d0/volumes" Sep 10 15:55:57.348040 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:57.348007 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:57.427868 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:57.427715 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:57.540582 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:57.540549 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:57.995427 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:57.995382 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:58.204967 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:58.204928 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:58.616719 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:55:58.616681 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:56:00.614431 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:00.614394 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:56:00.713299 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:00.713253 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:56:00.790075 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:00.790033 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:56:00.883891 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:00.883804 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:56:00.983915 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:00.983881 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:56:01.271611 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:01.271525 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:56:01.372290 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:01.372256 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:56:01.377526 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:01.377502 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0fcd7b0f-67df48d57d-x5bsn"] Sep 10 15:56:01.377933 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:01.377920 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d0e8c4b-e3a9-439c-9a19-c1fab80f54d0" containerName="payload-processing" Sep 10 15:56:01.377979 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:01.377935 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d0e8c4b-e3a9-439c-9a19-c1fab80f54d0" containerName="payload-processing" Sep 10 15:56:01.378025 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:01.378016 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d0e8c4b-e3a9-439c-9a19-c1fab80f54d0" containerName="payload-processing" Sep 10 15:56:01.381187 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:01.381168 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0fcd7b0f-67df48d57d-x5bsn" Sep 10 15:56:01.383334 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:01.383315 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-0fcd7b0f\"" Sep 10 15:56:01.393020 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:01.393000 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0fcd7b0f-67df48d57d-x5bsn"] Sep 10 15:56:01.448484 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:01.448442 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f697185-ddda-40a0-ba39-bf47c7d9f6e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-0fcd7b0f-67df48d57d-x5bsn\" (UID: \"1f697185-ddda-40a0-ba39-bf47c7d9f6e8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0fcd7b0f-67df48d57d-x5bsn" Sep 10 15:56:01.448484 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:01.448480 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klfb5\" (UniqueName: \"kubernetes.io/projected/1f697185-ddda-40a0-ba39-bf47c7d9f6e8-kube-api-access-klfb5\") pod \"payload-pre-processing-e2e-migrate-0fcd7b0f-67df48d57d-x5bsn\" (UID: \"1f697185-ddda-40a0-ba39-bf47c7d9f6e8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0fcd7b0f-67df48d57d-x5bsn" Sep 10 15:56:01.549263 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:01.549236 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f697185-ddda-40a0-ba39-bf47c7d9f6e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-0fcd7b0f-67df48d57d-x5bsn\" (UID: \"1f697185-ddda-40a0-ba39-bf47c7d9f6e8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0fcd7b0f-67df48d57d-x5bsn" Sep 10 15:56:01.549447 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:01.549272 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klfb5\" (UniqueName: \"kubernetes.io/projected/1f697185-ddda-40a0-ba39-bf47c7d9f6e8-kube-api-access-klfb5\") pod \"payload-pre-processing-e2e-migrate-0fcd7b0f-67df48d57d-x5bsn\" (UID: \"1f697185-ddda-40a0-ba39-bf47c7d9f6e8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0fcd7b0f-67df48d57d-x5bsn" Sep 10 15:56:01.550073 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:01.550049 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f697185-ddda-40a0-ba39-bf47c7d9f6e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-0fcd7b0f-67df48d57d-x5bsn\" (UID: \"1f697185-ddda-40a0-ba39-bf47c7d9f6e8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0fcd7b0f-67df48d57d-x5bsn" Sep 10 15:56:01.557209 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:01.557187 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klfb5\" (UniqueName: \"kubernetes.io/projected/1f697185-ddda-40a0-ba39-bf47c7d9f6e8-kube-api-access-klfb5\") pod \"payload-pre-processing-e2e-migrate-0fcd7b0f-67df48d57d-x5bsn\" (UID: \"1f697185-ddda-40a0-ba39-bf47c7d9f6e8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0fcd7b0f-67df48d57d-x5bsn" Sep 10 15:56:01.691477 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:01.691438 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0fcd7b0f-67df48d57d-x5bsn" Sep 10 15:56:01.962717 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:01.962646 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0fcd7b0f-67df48d57d-x5bsn"] Sep 10 15:56:01.965632 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:56:01.965599 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f697185_ddda_40a0_ba39_bf47c7d9f6e8.slice/crio-dd6217ed350f82cc6fe74103122678b41480d754138792fea52c1cf72ba7e35e WatchSource:0}: Error finding container dd6217ed350f82cc6fe74103122678b41480d754138792fea52c1cf72ba7e35e: Status 404 returned error can't find the container with id dd6217ed350f82cc6fe74103122678b41480d754138792fea52c1cf72ba7e35e Sep 10 15:56:02.101545 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:02.101518 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7" Sep 10 15:56:02.158821 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:02.158717 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bb806e3-78ee-40c8-8106-6d08b3063909-workload-socket\") pod \"6bb806e3-78ee-40c8-8106-6d08b3063909\" (UID: \"6bb806e3-78ee-40c8-8106-6d08b3063909\") " Sep 10 15:56:02.159011 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:02.158832 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bb806e3-78ee-40c8-8106-6d08b3063909-credential-socket\") pod \"6bb806e3-78ee-40c8-8106-6d08b3063909\" (UID: \"6bb806e3-78ee-40c8-8106-6d08b3063909\") " Sep 10 15:56:02.159011 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:02.158869 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bb806e3-78ee-40c8-8106-6d08b3063909-istio-token\") pod \"6bb806e3-78ee-40c8-8106-6d08b3063909\" (UID: \"6bb806e3-78ee-40c8-8106-6d08b3063909\") " Sep 10 15:56:02.159011 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:02.158897 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bb806e3-78ee-40c8-8106-6d08b3063909-workload-certs\") pod \"6bb806e3-78ee-40c8-8106-6d08b3063909\" (UID: \"6bb806e3-78ee-40c8-8106-6d08b3063909\") " Sep 10 15:56:02.159011 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:02.158941 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bb806e3-78ee-40c8-8106-6d08b3063909-istio-podinfo\") pod \"6bb806e3-78ee-40c8-8106-6d08b3063909\" (UID: \"6bb806e3-78ee-40c8-8106-6d08b3063909\") " Sep 10 15:56:02.159011 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:02.158967 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bb806e3-78ee-40c8-8106-6d08b3063909-istio-data\") pod \"6bb806e3-78ee-40c8-8106-6d08b3063909\" (UID: \"6bb806e3-78ee-40c8-8106-6d08b3063909\") " Sep 10 15:56:02.159011 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:02.158997 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bb806e3-78ee-40c8-8106-6d08b3063909-istiod-ca-cert\") pod \"6bb806e3-78ee-40c8-8106-6d08b3063909\" (UID: \"6bb806e3-78ee-40c8-8106-6d08b3063909\") " Sep 10 15:56:02.159385 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:02.159033 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bb806e3-78ee-40c8-8106-6d08b3063909-istio-envoy\") pod \"6bb806e3-78ee-40c8-8106-6d08b3063909\" (UID: \"6bb806e3-78ee-40c8-8106-6d08b3063909\") " Sep 10 15:56:02.159385 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:02.159080 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ssnrv\" (UniqueName: \"kubernetes.io/projected/6bb806e3-78ee-40c8-8106-6d08b3063909-kube-api-access-ssnrv\") pod \"6bb806e3-78ee-40c8-8106-6d08b3063909\" (UID: \"6bb806e3-78ee-40c8-8106-6d08b3063909\") " Sep 10 15:56:02.159505 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:02.158931 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bb806e3-78ee-40c8-8106-6d08b3063909-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6bb806e3-78ee-40c8-8106-6d08b3063909" (UID: "6bb806e3-78ee-40c8-8106-6d08b3063909"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:56:02.159573 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:02.159177 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bb806e3-78ee-40c8-8106-6d08b3063909-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6bb806e3-78ee-40c8-8106-6d08b3063909" (UID: "6bb806e3-78ee-40c8-8106-6d08b3063909"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:56:02.159573 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:02.159199 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bb806e3-78ee-40c8-8106-6d08b3063909-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6bb806e3-78ee-40c8-8106-6d08b3063909" (UID: "6bb806e3-78ee-40c8-8106-6d08b3063909"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:56:02.159573 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:02.159475 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bb806e3-78ee-40c8-8106-6d08b3063909-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6bb806e3-78ee-40c8-8106-6d08b3063909" (UID: "6bb806e3-78ee-40c8-8106-6d08b3063909"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:56:02.159884 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:02.159669 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bb806e3-78ee-40c8-8106-6d08b3063909-istio-data" (OuterVolumeSpecName: "istio-data") pod "6bb806e3-78ee-40c8-8106-6d08b3063909" (UID: "6bb806e3-78ee-40c8-8106-6d08b3063909"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:56:02.161808 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:02.161784 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bb806e3-78ee-40c8-8106-6d08b3063909-kube-api-access-ssnrv" (OuterVolumeSpecName: "kube-api-access-ssnrv") pod "6bb806e3-78ee-40c8-8106-6d08b3063909" (UID: "6bb806e3-78ee-40c8-8106-6d08b3063909"). InnerVolumeSpecName "kube-api-access-ssnrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:56:02.162038 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:02.162013 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6bb806e3-78ee-40c8-8106-6d08b3063909-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6bb806e3-78ee-40c8-8106-6d08b3063909" (UID: "6bb806e3-78ee-40c8-8106-6d08b3063909"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 15:56:02.162137 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:02.162075 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bb806e3-78ee-40c8-8106-6d08b3063909-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6bb806e3-78ee-40c8-8106-6d08b3063909" (UID: "6bb806e3-78ee-40c8-8106-6d08b3063909"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:56:02.162272 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:02.162249 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0fcd7b0f-67df48d57d-x5bsn" event={"ID":"1f697185-ddda-40a0-ba39-bf47c7d9f6e8","Type":"ContainerStarted","Data":"90e3512ea4f69c2da45ac3a6eb0cc22248a4f74e084a79af3d932b57bae96152"} Sep 10 15:56:02.162348 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:02.162331 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0fcd7b0f-67df48d57d-x5bsn" Sep 10 15:56:02.162396 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:02.162357 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0fcd7b0f-67df48d57d-x5bsn" event={"ID":"1f697185-ddda-40a0-ba39-bf47c7d9f6e8","Type":"ContainerStarted","Data":"dd6217ed350f82cc6fe74103122678b41480d754138792fea52c1cf72ba7e35e"} Sep 10 15:56:02.163685 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:02.163664 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bb806e3-78ee-40c8-8106-6d08b3063909-istio-token" (OuterVolumeSpecName: "istio-token") pod "6bb806e3-78ee-40c8-8106-6d08b3063909" (UID: "6bb806e3-78ee-40c8-8106-6d08b3063909"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:56:02.164097 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:02.164075 2576 generic.go:356] "Generic (PLEG): container finished" podID="6bb806e3-78ee-40c8-8106-6d08b3063909" containerID="29c94d5584368774a803470487b1d2ef7d43b912cd968c5310c4120fd027a10c" exitCode=0 Sep 10 15:56:02.164155 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:02.164118 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7" Sep 10 15:56:02.164155 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:02.164140 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7" event={"ID":"6bb806e3-78ee-40c8-8106-6d08b3063909","Type":"ContainerDied","Data":"29c94d5584368774a803470487b1d2ef7d43b912cd968c5310c4120fd027a10c"} Sep 10 15:56:02.164221 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:02.164191 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7" event={"ID":"6bb806e3-78ee-40c8-8106-6d08b3063909","Type":"ContainerDied","Data":"4f578a9eefb3d1e5cecc7e9a7afa9b150d117f89ea7e8341b715e2162899988e"} Sep 10 15:56:02.164255 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:02.164220 2576 scope.go:117] "RemoveContainer" containerID="29c94d5584368774a803470487b1d2ef7d43b912cd968c5310c4120fd027a10c" Sep 10 15:56:02.172927 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:02.172913 2576 scope.go:117] "RemoveContainer" containerID="8e61408b51cd872b3851de3bce527873fca78a630a92439413798bdcccecf5a0" Sep 10 15:56:02.177602 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:02.177555 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0fcd7b0f-67df48d57d-x5bsn" podStartSLOduration=1.177538758 podStartE2EDuration="1.177538758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:56:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:56:02.176321311 +0000 UTC m=+839.529840492" watchObservedRunningTime="2026-09-10 15:56:02.177538758 +0000 UTC m=+839.531057929" Sep 10 15:56:02.181669 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:02.181652 2576 scope.go:117] "RemoveContainer" containerID="29c94d5584368774a803470487b1d2ef7d43b912cd968c5310c4120fd027a10c" Sep 10 15:56:02.182009 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:56:02.181989 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29c94d5584368774a803470487b1d2ef7d43b912cd968c5310c4120fd027a10c\": container with ID starting with 29c94d5584368774a803470487b1d2ef7d43b912cd968c5310c4120fd027a10c not found: ID does not exist" containerID="29c94d5584368774a803470487b1d2ef7d43b912cd968c5310c4120fd027a10c" Sep 10 15:56:02.182104 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:02.182020 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29c94d5584368774a803470487b1d2ef7d43b912cd968c5310c4120fd027a10c"} err="failed to get container status \"29c94d5584368774a803470487b1d2ef7d43b912cd968c5310c4120fd027a10c\": rpc error: code = NotFound desc = could not find container \"29c94d5584368774a803470487b1d2ef7d43b912cd968c5310c4120fd027a10c\": container with ID starting with 29c94d5584368774a803470487b1d2ef7d43b912cd968c5310c4120fd027a10c not found: ID does not exist" Sep 10 15:56:02.182104 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:02.182042 2576 scope.go:117] "RemoveContainer" containerID="8e61408b51cd872b3851de3bce527873fca78a630a92439413798bdcccecf5a0" Sep 10 15:56:02.182322 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:56:02.182305 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e61408b51cd872b3851de3bce527873fca78a630a92439413798bdcccecf5a0\": container with ID starting with 8e61408b51cd872b3851de3bce527873fca78a630a92439413798bdcccecf5a0 not found: ID does not exist" containerID="8e61408b51cd872b3851de3bce527873fca78a630a92439413798bdcccecf5a0" Sep 10 15:56:02.182364 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:02.182327 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e61408b51cd872b3851de3bce527873fca78a630a92439413798bdcccecf5a0"} err="failed to get container status \"8e61408b51cd872b3851de3bce527873fca78a630a92439413798bdcccecf5a0\": rpc error: code = NotFound desc = could not find container \"8e61408b51cd872b3851de3bce527873fca78a630a92439413798bdcccecf5a0\": container with ID starting with 8e61408b51cd872b3851de3bce527873fca78a630a92439413798bdcccecf5a0 not found: ID does not exist" Sep 10 15:56:02.190364 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:02.190339 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7"] Sep 10 15:56:02.193663 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:02.193643 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-941c6070-openshift-default-6d86c55576-x2rb7"] Sep 10 15:56:02.260978 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:02.260947 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bb806e3-78ee-40c8-8106-6d08b3063909-istio-podinfo\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:56:02.260978 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:02.260977 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bb806e3-78ee-40c8-8106-6d08b3063909-istio-data\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:56:02.261178 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:02.260992 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bb806e3-78ee-40c8-8106-6d08b3063909-istiod-ca-cert\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:56:02.261178 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:02.261004 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bb806e3-78ee-40c8-8106-6d08b3063909-istio-envoy\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:56:02.261178 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:02.261015 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssnrv\" (UniqueName: \"kubernetes.io/projected/6bb806e3-78ee-40c8-8106-6d08b3063909-kube-api-access-ssnrv\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:56:02.261178 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:02.261030 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bb806e3-78ee-40c8-8106-6d08b3063909-workload-socket\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:56:02.261178 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:02.261043 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bb806e3-78ee-40c8-8106-6d08b3063909-credential-socket\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:56:02.261178 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:02.261055 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bb806e3-78ee-40c8-8106-6d08b3063909-istio-token\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:56:02.261178 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:02.261067 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bb806e3-78ee-40c8-8106-6d08b3063909-workload-certs\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:56:03.172599 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:03.172566 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bb806e3-78ee-40c8-8106-6d08b3063909" path="/var/lib/kubelet/pods/6bb806e3-78ee-40c8-8106-6d08b3063909/volumes" Sep 10 15:56:08.552386 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:08.552345 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:56:12.292000 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:12.291959 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 15:56:13.175375 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:13.175347 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0fcd7b0f-67df48d57d-x5bsn" Sep 10 15:56:13.926456 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:13.926426 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:56:13.987250 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:13.987212 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:56:25.530547 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:25.530513 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0fcd7b0f-j87gl"] Sep 10 15:56:25.530930 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:25.530908 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bb806e3-78ee-40c8-8106-6d08b3063909" containerName="istio-proxy" Sep 10 15:56:25.530930 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:25.530919 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bb806e3-78ee-40c8-8106-6d08b3063909" containerName="istio-proxy" Sep 10 15:56:25.531002 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:25.530998 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bb806e3-78ee-40c8-8106-6d08b3063909" containerName="istio-proxy" Sep 10 15:56:25.531038 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:25.531008 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bb806e3-78ee-40c8-8106-6d08b3063909" containerName="istio-proxy" Sep 10 15:56:25.536058 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:25.536038 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0fcd7b0f-j87gl" Sep 10 15:56:25.539194 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:25.539152 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 15:56:25.539722 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:25.539599 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwwht\"" Sep 10 15:56:25.541574 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:25.541554 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0fcd7b0f-j87gl"] Sep 10 15:56:25.574108 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:25.574086 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/451f8271-cfe9-4661-af10-ed613f14db31-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0fcd7b0f-j87gl\" (UID: \"451f8271-cfe9-4661-af10-ed613f14db31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0fcd7b0f-j87gl" Sep 10 15:56:25.675055 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:25.675028 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/451f8271-cfe9-4661-af10-ed613f14db31-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0fcd7b0f-j87gl\" (UID: \"451f8271-cfe9-4661-af10-ed613f14db31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0fcd7b0f-j87gl" Sep 10 15:56:25.675615 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:25.675594 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/451f8271-cfe9-4661-af10-ed613f14db31-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0fcd7b0f-j87gl\" (UID: \"451f8271-cfe9-4661-af10-ed613f14db31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0fcd7b0f-j87gl" Sep 10 15:56:25.847516 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:25.847490 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0fcd7b0f-j87gl" Sep 10 15:56:26.119405 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:26.119376 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0fcd7b0f-j87gl"] Sep 10 15:56:26.122241 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:56:26.122217 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod451f8271_cfe9_4661_af10_ed613f14db31.slice/crio-a2edc3ef553c8e333e6854b2bae7f44a6c8b87646fa00b21ede3da8b742b3dc8 WatchSource:0}: Error finding container a2edc3ef553c8e333e6854b2bae7f44a6c8b87646fa00b21ede3da8b742b3dc8: Status 404 returned error can't find the container with id a2edc3ef553c8e333e6854b2bae7f44a6c8b87646fa00b21ede3da8b742b3dc8 Sep 10 15:56:26.294185 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:26.294151 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0fcd7b0f-j87gl" event={"ID":"451f8271-cfe9-4661-af10-ed613f14db31","Type":"ContainerStarted","Data":"a2edc3ef553c8e333e6854b2bae7f44a6c8b87646fa00b21ede3da8b742b3dc8"} Sep 10 15:56:29.312205 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:29.312167 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0fcd7b0f-j87gl" event={"ID":"451f8271-cfe9-4661-af10-ed613f14db31","Type":"ContainerStarted","Data":"273587ce5455ed2b0a3ec21238d2e1c13e2c8b6769d381dd4613cf64cc3d689b"} Sep 10 15:56:29.327197 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:29.327155 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0fcd7b0f-j87gl" podStartSLOduration=1.658918934 podStartE2EDuration="4.327140918s" totalImagesPullingTime="2.668221984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:56:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:56:29.326631222 +0000 UTC m=+866.680150407" watchObservedRunningTime="2026-09-10 15:56:29.327140918 +0000 UTC m=+866.680660078" Sep 10 15:56:30.319756 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:30.319693 2576 generic.go:356] "Generic (PLEG): container finished" podID="451f8271-cfe9-4661-af10-ed613f14db31" containerID="273587ce5455ed2b0a3ec21238d2e1c13e2c8b6769d381dd4613cf64cc3d689b" exitCode=0 Sep 10 15:56:30.320144 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:30.319752 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0fcd7b0f-j87gl" event={"ID":"451f8271-cfe9-4661-af10-ed613f14db31","Type":"ContainerDied","Data":"273587ce5455ed2b0a3ec21238d2e1c13e2c8b6769d381dd4613cf64cc3d689b"} Sep 10 15:56:31.464326 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:31.464300 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0fcd7b0f-j87gl" Sep 10 15:56:31.525100 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:31.525074 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/451f8271-cfe9-4661-af10-ed613f14db31-maas-api-service-ca\") pod \"451f8271-cfe9-4661-af10-ed613f14db31\" (UID: \"451f8271-cfe9-4661-af10-ed613f14db31\") " Sep 10 15:56:31.525384 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:31.525361 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/451f8271-cfe9-4661-af10-ed613f14db31-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "451f8271-cfe9-4661-af10-ed613f14db31" (UID: "451f8271-cfe9-4661-af10-ed613f14db31"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:56:31.626868 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:31.626818 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/451f8271-cfe9-4661-af10-ed613f14db31-maas-api-service-ca\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:56:32.330069 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:32.330042 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0fcd7b0f-j87gl" Sep 10 15:56:32.330242 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:32.330085 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0fcd7b0f-j87gl" event={"ID":"451f8271-cfe9-4661-af10-ed613f14db31","Type":"ContainerDied","Data":"a2edc3ef553c8e333e6854b2bae7f44a6c8b87646fa00b21ede3da8b742b3dc8"} Sep 10 15:56:32.330242 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:32.330122 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2edc3ef553c8e333e6854b2bae7f44a6c8b87646fa00b21ede3da8b742b3dc8" Sep 10 15:56:32.781003 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:32.780933 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 15:56:38.542008 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:38.541973 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:56:40.986696 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:40.986662 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-314824-nl57z"] Sep 10 15:56:40.987194 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:40.987107 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bb806e3-78ee-40c8-8106-6d08b3063909" containerName="istio-proxy" Sep 10 15:56:40.987194 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:40.987119 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bb806e3-78ee-40c8-8106-6d08b3063909" containerName="istio-proxy" Sep 10 15:56:40.987194 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:40.987137 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="451f8271-cfe9-4661-af10-ed613f14db31" containerName="revoke-keys" Sep 10 15:56:40.987194 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:40.987142 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="451f8271-cfe9-4661-af10-ed613f14db31" containerName="revoke-keys" Sep 10 15:56:40.987403 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:40.987198 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="451f8271-cfe9-4661-af10-ed613f14db31" containerName="revoke-keys" Sep 10 15:56:40.990780 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:40.990758 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-314824-nl57z" Sep 10 15:56:40.993409 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:40.993388 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwwht\"" Sep 10 15:56:40.993514 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:40.993493 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 15:56:40.995183 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:40.995156 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-314824-nl57z"] Sep 10 15:56:41.124131 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:41.124095 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d833c84-274e-46e6-9920-7b1c8f80f748-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-314824-nl57z\" (UID: \"9d833c84-274e-46e6-9920-7b1c8f80f748\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-314824-nl57z" Sep 10 15:56:41.224128 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:41.224095 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:56:41.224727 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:41.224708 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d833c84-274e-46e6-9920-7b1c8f80f748-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-314824-nl57z\" (UID: \"9d833c84-274e-46e6-9920-7b1c8f80f748\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-314824-nl57z" Sep 10 15:56:41.225314 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:41.225292 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d833c84-274e-46e6-9920-7b1c8f80f748-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-314824-nl57z\" (UID: \"9d833c84-274e-46e6-9920-7b1c8f80f748\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-314824-nl57z" Sep 10 15:56:41.305150 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:41.305117 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-314824-nl57z" Sep 10 15:56:41.572037 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:41.572015 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-314824-nl57z"] Sep 10 15:56:41.574282 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:56:41.574256 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d833c84_274e_46e6_9920_7b1c8f80f748.slice/crio-8e51a71380f7f721b8f5f10b280ce0d5b0d669a05f2c7825434c39f59721c67d WatchSource:0}: Error finding container 8e51a71380f7f721b8f5f10b280ce0d5b0d669a05f2c7825434c39f59721c67d: Status 404 returned error can't find the container with id 8e51a71380f7f721b8f5f10b280ce0d5b0d669a05f2c7825434c39f59721c67d Sep 10 15:56:41.938759 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:41.938715 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:56:41.966337 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:41.966306 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0fcd7b0f-67df48d57d-x5bsn"] Sep 10 15:56:41.966575 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:41.966541 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0fcd7b0f-67df48d57d-x5bsn" podUID="1f697185-ddda-40a0-ba39-bf47c7d9f6e8" containerName="payload-pre-processing" containerID="cri-o://90e3512ea4f69c2da45ac3a6eb0cc22248a4f74e084a79af3d932b57bae96152" gracePeriod=30 Sep 10 15:56:42.038516 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:42.038481 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:56:42.139317 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:42.139056 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:56:42.167136 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:42.167067 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0fcd7b0f-j87gl"] Sep 10 15:56:42.169012 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:42.168992 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0fcd7b0f-j87gl"] Sep 10 15:56:42.210537 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:42.210079 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0fcd7b0f-67df48d57d-x5bsn" Sep 10 15:56:42.235253 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:42.235221 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:56:42.308366 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:42.308333 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:56:42.336022 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:42.335996 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f697185-ddda-40a0-ba39-bf47c7d9f6e8-plugins-config-volume\") pod \"1f697185-ddda-40a0-ba39-bf47c7d9f6e8\" (UID: \"1f697185-ddda-40a0-ba39-bf47c7d9f6e8\") " Sep 10 15:56:42.336149 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:42.336087 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-klfb5\" (UniqueName: \"kubernetes.io/projected/1f697185-ddda-40a0-ba39-bf47c7d9f6e8-kube-api-access-klfb5\") pod \"1f697185-ddda-40a0-ba39-bf47c7d9f6e8\" (UID: \"1f697185-ddda-40a0-ba39-bf47c7d9f6e8\") " Sep 10 15:56:42.336423 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:42.336399 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f697185-ddda-40a0-ba39-bf47c7d9f6e8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1f697185-ddda-40a0-ba39-bf47c7d9f6e8" (UID: "1f697185-ddda-40a0-ba39-bf47c7d9f6e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:56:42.338194 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:42.338173 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f697185-ddda-40a0-ba39-bf47c7d9f6e8-kube-api-access-klfb5" (OuterVolumeSpecName: "kube-api-access-klfb5") pod "1f697185-ddda-40a0-ba39-bf47c7d9f6e8" (UID: "1f697185-ddda-40a0-ba39-bf47c7d9f6e8"). InnerVolumeSpecName "kube-api-access-klfb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:56:42.386480 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:42.386443 2576 generic.go:356] "Generic (PLEG): container finished" podID="1f697185-ddda-40a0-ba39-bf47c7d9f6e8" containerID="90e3512ea4f69c2da45ac3a6eb0cc22248a4f74e084a79af3d932b57bae96152" exitCode=0 Sep 10 15:56:42.386673 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:42.386623 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0fcd7b0f-67df48d57d-x5bsn" event={"ID":"1f697185-ddda-40a0-ba39-bf47c7d9f6e8","Type":"ContainerDied","Data":"90e3512ea4f69c2da45ac3a6eb0cc22248a4f74e084a79af3d932b57bae96152"} Sep 10 15:56:42.386776 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:42.386683 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0fcd7b0f-67df48d57d-x5bsn" event={"ID":"1f697185-ddda-40a0-ba39-bf47c7d9f6e8","Type":"ContainerDied","Data":"dd6217ed350f82cc6fe74103122678b41480d754138792fea52c1cf72ba7e35e"} Sep 10 15:56:42.386776 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:42.386715 2576 scope.go:117] "RemoveContainer" containerID="90e3512ea4f69c2da45ac3a6eb0cc22248a4f74e084a79af3d932b57bae96152" Sep 10 15:56:42.386942 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:42.386921 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0fcd7b0f-67df48d57d-x5bsn" Sep 10 15:56:42.390319 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:42.390288 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-314824-nl57z" event={"ID":"9d833c84-274e-46e6-9920-7b1c8f80f748","Type":"ContainerStarted","Data":"58db7d05c5a033825a8ceb7369e5344fa79737cc9f28b2936870b7548c30a855"} Sep 10 15:56:42.390432 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:42.390339 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-314824-nl57z" event={"ID":"9d833c84-274e-46e6-9920-7b1c8f80f748","Type":"ContainerStarted","Data":"8e51a71380f7f721b8f5f10b280ce0d5b0d669a05f2c7825434c39f59721c67d"} Sep 10 15:56:42.407794 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:42.405457 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-314824-nl57z" podStartSLOduration=2.405435593 podStartE2EDuration="2.405435593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:56:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:56:42.404178741 +0000 UTC m=+879.757697930" watchObservedRunningTime="2026-09-10 15:56:42.405435593 +0000 UTC m=+879.758954753" Sep 10 15:56:42.407794 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:42.406089 2576 scope.go:117] "RemoveContainer" containerID="90e3512ea4f69c2da45ac3a6eb0cc22248a4f74e084a79af3d932b57bae96152" Sep 10 15:56:42.407794 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:56:42.406524 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90e3512ea4f69c2da45ac3a6eb0cc22248a4f74e084a79af3d932b57bae96152\": container with ID starting with 90e3512ea4f69c2da45ac3a6eb0cc22248a4f74e084a79af3d932b57bae96152 not found: ID does not exist" containerID="90e3512ea4f69c2da45ac3a6eb0cc22248a4f74e084a79af3d932b57bae96152" Sep 10 15:56:42.407794 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:42.406558 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90e3512ea4f69c2da45ac3a6eb0cc22248a4f74e084a79af3d932b57bae96152"} err="failed to get container status \"90e3512ea4f69c2da45ac3a6eb0cc22248a4f74e084a79af3d932b57bae96152\": rpc error: code = NotFound desc = could not find container \"90e3512ea4f69c2da45ac3a6eb0cc22248a4f74e084a79af3d932b57bae96152\": container with ID starting with 90e3512ea4f69c2da45ac3a6eb0cc22248a4f74e084a79af3d932b57bae96152 not found: ID does not exist" Sep 10 15:56:42.417963 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:42.417886 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0fcd7b0f-67df48d57d-x5bsn"] Sep 10 15:56:42.421622 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:42.421598 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0fcd7b0f-67df48d57d-x5bsn"] Sep 10 15:56:42.437215 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:42.437194 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klfb5\" (UniqueName: \"kubernetes.io/projected/1f697185-ddda-40a0-ba39-bf47c7d9f6e8-kube-api-access-klfb5\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:56:42.437319 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:42.437217 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f697185-ddda-40a0-ba39-bf47c7d9f6e8-plugins-config-volume\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:56:42.759839 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:42.759706 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:56:42.984352 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:42.984316 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-529c3efe-openshift-default-5b4478f6-595rf"] Sep 10 15:56:42.984935 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:42.984913 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f697185-ddda-40a0-ba39-bf47c7d9f6e8" containerName="payload-pre-processing" Sep 10 15:56:42.984935 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:42.984935 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f697185-ddda-40a0-ba39-bf47c7d9f6e8" containerName="payload-pre-processing" Sep 10 15:56:42.985145 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:42.985070 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f697185-ddda-40a0-ba39-bf47c7d9f6e8" containerName="payload-pre-processing" Sep 10 15:56:42.988183 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:42.988162 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-529c3efe-openshift-default-5b4478f6-595rf" Sep 10 15:56:42.990334 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:42.990313 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-529c3efe-openshift-default-dockercfg-psx4x\"" Sep 10 15:56:42.998138 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:42.998112 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-529c3efe-openshift-default-5b4478f6-595rf"] Sep 10 15:56:43.004660 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:43.004631 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:56:43.145067 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:43.145030 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/780ebc58-fc9c-46d3-b893-6bff1e70a445-istio-token\") pod \"e2e-ait-529c3efe-openshift-default-5b4478f6-595rf\" (UID: \"780ebc58-fc9c-46d3-b893-6bff1e70a445\") " pod="openshift-ingress/e2e-ait-529c3efe-openshift-default-5b4478f6-595rf" Sep 10 15:56:43.145445 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:43.145075 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/780ebc58-fc9c-46d3-b893-6bff1e70a445-istiod-ca-cert\") pod \"e2e-ait-529c3efe-openshift-default-5b4478f6-595rf\" (UID: \"780ebc58-fc9c-46d3-b893-6bff1e70a445\") " pod="openshift-ingress/e2e-ait-529c3efe-openshift-default-5b4478f6-595rf" Sep 10 15:56:43.145445 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:43.145106 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq6b4\" (UniqueName: \"kubernetes.io/projected/780ebc58-fc9c-46d3-b893-6bff1e70a445-kube-api-access-pq6b4\") pod \"e2e-ait-529c3efe-openshift-default-5b4478f6-595rf\" (UID: \"780ebc58-fc9c-46d3-b893-6bff1e70a445\") " pod="openshift-ingress/e2e-ait-529c3efe-openshift-default-5b4478f6-595rf" Sep 10 15:56:43.145445 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:43.145140 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/780ebc58-fc9c-46d3-b893-6bff1e70a445-workload-socket\") pod \"e2e-ait-529c3efe-openshift-default-5b4478f6-595rf\" (UID: \"780ebc58-fc9c-46d3-b893-6bff1e70a445\") " pod="openshift-ingress/e2e-ait-529c3efe-openshift-default-5b4478f6-595rf" Sep 10 15:56:43.145445 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:43.145174 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/780ebc58-fc9c-46d3-b893-6bff1e70a445-istio-podinfo\") pod \"e2e-ait-529c3efe-openshift-default-5b4478f6-595rf\" (UID: \"780ebc58-fc9c-46d3-b893-6bff1e70a445\") " pod="openshift-ingress/e2e-ait-529c3efe-openshift-default-5b4478f6-595rf" Sep 10 15:56:43.145445 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:43.145222 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/780ebc58-fc9c-46d3-b893-6bff1e70a445-credential-socket\") pod \"e2e-ait-529c3efe-openshift-default-5b4478f6-595rf\" (UID: \"780ebc58-fc9c-46d3-b893-6bff1e70a445\") " pod="openshift-ingress/e2e-ait-529c3efe-openshift-default-5b4478f6-595rf" Sep 10 15:56:43.145445 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:43.145242 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/780ebc58-fc9c-46d3-b893-6bff1e70a445-istio-data\") pod \"e2e-ait-529c3efe-openshift-default-5b4478f6-595rf\" (UID: \"780ebc58-fc9c-46d3-b893-6bff1e70a445\") " pod="openshift-ingress/e2e-ait-529c3efe-openshift-default-5b4478f6-595rf" Sep 10 15:56:43.145445 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:43.145310 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/780ebc58-fc9c-46d3-b893-6bff1e70a445-istio-envoy\") pod \"e2e-ait-529c3efe-openshift-default-5b4478f6-595rf\" (UID: \"780ebc58-fc9c-46d3-b893-6bff1e70a445\") " pod="openshift-ingress/e2e-ait-529c3efe-openshift-default-5b4478f6-595rf" Sep 10 15:56:43.145445 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:43.145337 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/780ebc58-fc9c-46d3-b893-6bff1e70a445-workload-certs\") pod \"e2e-ait-529c3efe-openshift-default-5b4478f6-595rf\" (UID: \"780ebc58-fc9c-46d3-b893-6bff1e70a445\") " pod="openshift-ingress/e2e-ait-529c3efe-openshift-default-5b4478f6-595rf" Sep 10 15:56:43.169045 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:43.169005 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f697185-ddda-40a0-ba39-bf47c7d9f6e8" path="/var/lib/kubelet/pods/1f697185-ddda-40a0-ba39-bf47c7d9f6e8/volumes" Sep 10 15:56:43.169400 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:43.169381 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="451f8271-cfe9-4661-af10-ed613f14db31" path="/var/lib/kubelet/pods/451f8271-cfe9-4661-af10-ed613f14db31/volumes" Sep 10 15:56:43.246093 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:43.246040 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/780ebc58-fc9c-46d3-b893-6bff1e70a445-istio-data\") pod \"e2e-ait-529c3efe-openshift-default-5b4478f6-595rf\" (UID: \"780ebc58-fc9c-46d3-b893-6bff1e70a445\") " pod="openshift-ingress/e2e-ait-529c3efe-openshift-default-5b4478f6-595rf" Sep 10 15:56:43.246284 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:43.246132 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/780ebc58-fc9c-46d3-b893-6bff1e70a445-istio-envoy\") pod \"e2e-ait-529c3efe-openshift-default-5b4478f6-595rf\" (UID: \"780ebc58-fc9c-46d3-b893-6bff1e70a445\") " pod="openshift-ingress/e2e-ait-529c3efe-openshift-default-5b4478f6-595rf" Sep 10 15:56:43.246284 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:43.246173 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/780ebc58-fc9c-46d3-b893-6bff1e70a445-workload-certs\") pod \"e2e-ait-529c3efe-openshift-default-5b4478f6-595rf\" (UID: \"780ebc58-fc9c-46d3-b893-6bff1e70a445\") " pod="openshift-ingress/e2e-ait-529c3efe-openshift-default-5b4478f6-595rf" Sep 10 15:56:43.246284 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:43.246251 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/780ebc58-fc9c-46d3-b893-6bff1e70a445-istio-token\") pod \"e2e-ait-529c3efe-openshift-default-5b4478f6-595rf\" (UID: \"780ebc58-fc9c-46d3-b893-6bff1e70a445\") " pod="openshift-ingress/e2e-ait-529c3efe-openshift-default-5b4478f6-595rf" Sep 10 15:56:43.246472 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:43.246284 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/780ebc58-fc9c-46d3-b893-6bff1e70a445-istiod-ca-cert\") pod \"e2e-ait-529c3efe-openshift-default-5b4478f6-595rf\" (UID: \"780ebc58-fc9c-46d3-b893-6bff1e70a445\") " pod="openshift-ingress/e2e-ait-529c3efe-openshift-default-5b4478f6-595rf" Sep 10 15:56:43.246472 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:43.246314 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq6b4\" (UniqueName: \"kubernetes.io/projected/780ebc58-fc9c-46d3-b893-6bff1e70a445-kube-api-access-pq6b4\") pod \"e2e-ait-529c3efe-openshift-default-5b4478f6-595rf\" (UID: \"780ebc58-fc9c-46d3-b893-6bff1e70a445\") " pod="openshift-ingress/e2e-ait-529c3efe-openshift-default-5b4478f6-595rf" Sep 10 15:56:43.246472 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:43.246342 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/780ebc58-fc9c-46d3-b893-6bff1e70a445-workload-socket\") pod \"e2e-ait-529c3efe-openshift-default-5b4478f6-595rf\" (UID: \"780ebc58-fc9c-46d3-b893-6bff1e70a445\") " pod="openshift-ingress/e2e-ait-529c3efe-openshift-default-5b4478f6-595rf" Sep 10 15:56:43.246472 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:43.246383 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/780ebc58-fc9c-46d3-b893-6bff1e70a445-istio-podinfo\") pod \"e2e-ait-529c3efe-openshift-default-5b4478f6-595rf\" (UID: \"780ebc58-fc9c-46d3-b893-6bff1e70a445\") " pod="openshift-ingress/e2e-ait-529c3efe-openshift-default-5b4478f6-595rf" Sep 10 15:56:43.246472 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:43.246421 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/780ebc58-fc9c-46d3-b893-6bff1e70a445-credential-socket\") pod \"e2e-ait-529c3efe-openshift-default-5b4478f6-595rf\" (UID: \"780ebc58-fc9c-46d3-b893-6bff1e70a445\") " pod="openshift-ingress/e2e-ait-529c3efe-openshift-default-5b4478f6-595rf" Sep 10 15:56:43.247103 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:43.247073 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/780ebc58-fc9c-46d3-b893-6bff1e70a445-workload-certs\") pod \"e2e-ait-529c3efe-openshift-default-5b4478f6-595rf\" (UID: \"780ebc58-fc9c-46d3-b893-6bff1e70a445\") " pod="openshift-ingress/e2e-ait-529c3efe-openshift-default-5b4478f6-595rf" Sep 10 15:56:43.247211 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:43.247138 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/780ebc58-fc9c-46d3-b893-6bff1e70a445-credential-socket\") pod \"e2e-ait-529c3efe-openshift-default-5b4478f6-595rf\" (UID: \"780ebc58-fc9c-46d3-b893-6bff1e70a445\") " pod="openshift-ingress/e2e-ait-529c3efe-openshift-default-5b4478f6-595rf" Sep 10 15:56:43.247211 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:43.247174 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/780ebc58-fc9c-46d3-b893-6bff1e70a445-workload-socket\") pod \"e2e-ait-529c3efe-openshift-default-5b4478f6-595rf\" (UID: \"780ebc58-fc9c-46d3-b893-6bff1e70a445\") " pod="openshift-ingress/e2e-ait-529c3efe-openshift-default-5b4478f6-595rf" Sep 10 15:56:43.247307 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:43.247212 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/780ebc58-fc9c-46d3-b893-6bff1e70a445-istio-data\") pod \"e2e-ait-529c3efe-openshift-default-5b4478f6-595rf\" (UID: \"780ebc58-fc9c-46d3-b893-6bff1e70a445\") " pod="openshift-ingress/e2e-ait-529c3efe-openshift-default-5b4478f6-595rf" Sep 10 15:56:43.247373 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:43.247351 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/780ebc58-fc9c-46d3-b893-6bff1e70a445-istiod-ca-cert\") pod \"e2e-ait-529c3efe-openshift-default-5b4478f6-595rf\" (UID: \"780ebc58-fc9c-46d3-b893-6bff1e70a445\") " pod="openshift-ingress/e2e-ait-529c3efe-openshift-default-5b4478f6-595rf" Sep 10 15:56:43.249049 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:43.249028 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/780ebc58-fc9c-46d3-b893-6bff1e70a445-istio-envoy\") pod \"e2e-ait-529c3efe-openshift-default-5b4478f6-595rf\" (UID: \"780ebc58-fc9c-46d3-b893-6bff1e70a445\") " pod="openshift-ingress/e2e-ait-529c3efe-openshift-default-5b4478f6-595rf" Sep 10 15:56:43.249202 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:43.249183 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/780ebc58-fc9c-46d3-b893-6bff1e70a445-istio-podinfo\") pod \"e2e-ait-529c3efe-openshift-default-5b4478f6-595rf\" (UID: \"780ebc58-fc9c-46d3-b893-6bff1e70a445\") " pod="openshift-ingress/e2e-ait-529c3efe-openshift-default-5b4478f6-595rf" Sep 10 15:56:43.256948 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:43.256921 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/780ebc58-fc9c-46d3-b893-6bff1e70a445-istio-token\") pod \"e2e-ait-529c3efe-openshift-default-5b4478f6-595rf\" (UID: \"780ebc58-fc9c-46d3-b893-6bff1e70a445\") " pod="openshift-ingress/e2e-ait-529c3efe-openshift-default-5b4478f6-595rf" Sep 10 15:56:43.257100 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:43.257078 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq6b4\" (UniqueName: \"kubernetes.io/projected/780ebc58-fc9c-46d3-b893-6bff1e70a445-kube-api-access-pq6b4\") pod \"e2e-ait-529c3efe-openshift-default-5b4478f6-595rf\" (UID: \"780ebc58-fc9c-46d3-b893-6bff1e70a445\") " pod="openshift-ingress/e2e-ait-529c3efe-openshift-default-5b4478f6-595rf" Sep 10 15:56:43.288890 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:43.288857 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:56:43.300160 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:43.300143 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-529c3efe-openshift-default-5b4478f6-595rf" Sep 10 15:56:43.399827 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:43.399753 2576 generic.go:356] "Generic (PLEG): container finished" podID="9d833c84-274e-46e6-9920-7b1c8f80f748" containerID="58db7d05c5a033825a8ceb7369e5344fa79737cc9f28b2936870b7548c30a855" exitCode=0 Sep 10 15:56:43.399827 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:43.399810 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-314824-nl57z" event={"ID":"9d833c84-274e-46e6-9920-7b1c8f80f748","Type":"ContainerDied","Data":"58db7d05c5a033825a8ceb7369e5344fa79737cc9f28b2936870b7548c30a855"} Sep 10 15:56:43.584558 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:43.584530 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-529c3efe-openshift-default-5b4478f6-595rf"] Sep 10 15:56:43.586179 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:56:43.586150 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod780ebc58_fc9c_46d3_b893_6bff1e70a445.slice/crio-cdffbb030f17767f6e0f23a24796dd580191e2177880d1136bb5a9c7d584c0df WatchSource:0}: Error finding container cdffbb030f17767f6e0f23a24796dd580191e2177880d1136bb5a9c7d584c0df: Status 404 returned error can't find the container with id cdffbb030f17767f6e0f23a24796dd580191e2177880d1136bb5a9c7d584c0df Sep 10 15:56:43.588519 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:43.588483 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 15:56:43.588624 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:43.588551 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 15:56:43.588624 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:43.588580 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 15:56:44.409184 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:44.409132 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-529c3efe-openshift-default-5b4478f6-595rf" event={"ID":"780ebc58-fc9c-46d3-b893-6bff1e70a445","Type":"ContainerStarted","Data":"ad2ffb9e031da286e9611cbe501ca978e75463d0e82ed4a79a67c378b6404011"} Sep 10 15:56:44.409685 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:44.409217 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-529c3efe-openshift-default-5b4478f6-595rf" event={"ID":"780ebc58-fc9c-46d3-b893-6bff1e70a445","Type":"ContainerStarted","Data":"cdffbb030f17767f6e0f23a24796dd580191e2177880d1136bb5a9c7d584c0df"} Sep 10 15:56:44.429425 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:44.429374 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-529c3efe-openshift-default-5b4478f6-595rf" podStartSLOduration=2.429360211 podStartE2EDuration="2.429360211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:56:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:56:44.427626083 +0000 UTC m=+881.781145248" watchObservedRunningTime="2026-09-10 15:56:44.429360211 +0000 UTC m=+881.782879369" Sep 10 15:56:44.550023 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:44.549999 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-314824-nl57z" Sep 10 15:56:44.660951 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:44.660867 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d833c84-274e-46e6-9920-7b1c8f80f748-maas-api-service-ca\") pod \"9d833c84-274e-46e6-9920-7b1c8f80f748\" (UID: \"9d833c84-274e-46e6-9920-7b1c8f80f748\") " Sep 10 15:56:44.661282 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:44.661257 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d833c84-274e-46e6-9920-7b1c8f80f748-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9d833c84-274e-46e6-9920-7b1c8f80f748" (UID: "9d833c84-274e-46e6-9920-7b1c8f80f748"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:56:44.762475 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:44.762442 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d833c84-274e-46e6-9920-7b1c8f80f748-maas-api-service-ca\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:56:45.300376 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:45.300338 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-529c3efe-openshift-default-5b4478f6-595rf" Sep 10 15:56:45.307747 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:45.307712 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-529c3efe-openshift-default-5b4478f6-595rf" Sep 10 15:56:45.417224 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:45.417183 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-314824-nl57z" event={"ID":"9d833c84-274e-46e6-9920-7b1c8f80f748","Type":"ContainerDied","Data":"8e51a71380f7f721b8f5f10b280ce0d5b0d669a05f2c7825434c39f59721c67d"} Sep 10 15:56:45.417224 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:45.417216 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-314824-nl57z" Sep 10 15:56:45.417757 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:45.417246 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e51a71380f7f721b8f5f10b280ce0d5b0d669a05f2c7825434c39f59721c67d" Sep 10 15:56:45.417757 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:45.417470 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-529c3efe-openshift-default-5b4478f6-595rf" Sep 10 15:56:45.418886 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:45.418866 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-529c3efe-openshift-default-5b4478f6-595rf" Sep 10 15:56:45.834527 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:45.834492 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:56:45.939398 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:45.939354 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:56:47.079184 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:47.079152 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:56:47.231593 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:47.231543 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:56:47.329709 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:47.329626 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:56:47.648799 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:47.648690 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:56:47.744535 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:47.744498 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:56:47.828385 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:47.828352 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-529c3efe-86474f4cd-4zqr6"] Sep 10 15:56:47.828792 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:47.828772 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d833c84-274e-46e6-9920-7b1c8f80f748" containerName="revoke-keys" Sep 10 15:56:47.828792 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:47.828790 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d833c84-274e-46e6-9920-7b1c8f80f748" containerName="revoke-keys" Sep 10 15:56:47.828950 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:47.828881 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d833c84-274e-46e6-9920-7b1c8f80f748" containerName="revoke-keys" Sep 10 15:56:47.832085 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:47.832068 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-529c3efe-86474f4cd-4zqr6" Sep 10 15:56:47.834364 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:47.834349 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-529c3efe\"" Sep 10 15:56:47.834452 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:47.834363 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-529c3efe-dockercfg-g2xqq\"" Sep 10 15:56:47.841008 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:47.840986 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-529c3efe-86474f4cd-4zqr6"] Sep 10 15:56:47.993084 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:47.993057 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gqmb\" (UniqueName: \"kubernetes.io/projected/7143c83a-cf64-40d8-a3f1-8dc1ec11803b-kube-api-access-8gqmb\") pod \"payload-pre-processing-e2e-ait-529c3efe-86474f4cd-4zqr6\" (UID: \"7143c83a-cf64-40d8-a3f1-8dc1ec11803b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-529c3efe-86474f4cd-4zqr6" Sep 10 15:56:47.993205 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:47.993099 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7143c83a-cf64-40d8-a3f1-8dc1ec11803b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-529c3efe-86474f4cd-4zqr6\" (UID: \"7143c83a-cf64-40d8-a3f1-8dc1ec11803b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-529c3efe-86474f4cd-4zqr6" Sep 10 15:56:48.093882 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:48.093853 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gqmb\" (UniqueName: \"kubernetes.io/projected/7143c83a-cf64-40d8-a3f1-8dc1ec11803b-kube-api-access-8gqmb\") pod \"payload-pre-processing-e2e-ait-529c3efe-86474f4cd-4zqr6\" (UID: \"7143c83a-cf64-40d8-a3f1-8dc1ec11803b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-529c3efe-86474f4cd-4zqr6" Sep 10 15:56:48.094247 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:48.093897 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7143c83a-cf64-40d8-a3f1-8dc1ec11803b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-529c3efe-86474f4cd-4zqr6\" (UID: \"7143c83a-cf64-40d8-a3f1-8dc1ec11803b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-529c3efe-86474f4cd-4zqr6" Sep 10 15:56:48.094485 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:48.094467 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7143c83a-cf64-40d8-a3f1-8dc1ec11803b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-529c3efe-86474f4cd-4zqr6\" (UID: \"7143c83a-cf64-40d8-a3f1-8dc1ec11803b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-529c3efe-86474f4cd-4zqr6" Sep 10 15:56:48.108869 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:48.108851 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gqmb\" (UniqueName: \"kubernetes.io/projected/7143c83a-cf64-40d8-a3f1-8dc1ec11803b-kube-api-access-8gqmb\") pod \"payload-pre-processing-e2e-ait-529c3efe-86474f4cd-4zqr6\" (UID: \"7143c83a-cf64-40d8-a3f1-8dc1ec11803b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-529c3efe-86474f4cd-4zqr6" Sep 10 15:56:48.173045 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:48.173026 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-529c3efe-86474f4cd-4zqr6" Sep 10 15:56:48.442436 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:48.442410 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-529c3efe-86474f4cd-4zqr6"] Sep 10 15:56:48.444156 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:56:48.444133 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7143c83a_cf64_40d8_a3f1_8dc1ec11803b.slice/crio-0eae1771126a54989a8e4a6d2385d1591697675711728b7dcdc1f94a39773e4e WatchSource:0}: Error finding container 0eae1771126a54989a8e4a6d2385d1591697675711728b7dcdc1f94a39773e4e: Status 404 returned error can't find the container with id 0eae1771126a54989a8e4a6d2385d1591697675711728b7dcdc1f94a39773e4e Sep 10 15:56:49.439466 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:49.439427 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-529c3efe-86474f4cd-4zqr6" event={"ID":"7143c83a-cf64-40d8-a3f1-8dc1ec11803b","Type":"ContainerStarted","Data":"d3b1d8f7b1f934bb6e20f49718b4a132292fb5cee71a7477407126e18f6809b2"} Sep 10 15:56:49.439854 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:49.439484 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-529c3efe-86474f4cd-4zqr6" event={"ID":"7143c83a-cf64-40d8-a3f1-8dc1ec11803b","Type":"ContainerStarted","Data":"0eae1771126a54989a8e4a6d2385d1591697675711728b7dcdc1f94a39773e4e"} Sep 10 15:56:49.439854 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:49.439510 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-529c3efe-86474f4cd-4zqr6" Sep 10 15:56:49.455321 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:49.455273 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-529c3efe-86474f4cd-4zqr6" podStartSLOduration=2.455259771 podStartE2EDuration="2.455259771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:56:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:56:49.452502622 +0000 UTC m=+886.806021804" watchObservedRunningTime="2026-09-10 15:56:49.455259771 +0000 UTC m=+886.808778932" Sep 10 15:56:51.233568 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:51.233533 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:56:51.943687 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:51.943647 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:56:52.678307 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:52.678247 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:56:52.689087 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:52.689057 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-314824-55c4c7d84f-szjbr"] Sep 10 15:56:52.689318 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:52.689292 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-314824-55c4c7d84f-szjbr" podUID="e5a17db3-3e2c-421a-bacf-7b9844a8a388" containerName="payload-processing" containerID="cri-o://f9bf73f298e8d429882167b9e56b5b3dfa6937d108e152c2fec14251a597fcd2" gracePeriod=30 Sep 10 15:56:52.780203 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:52.780171 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:56:52.874704 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:52.874658 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-314824-nl57z"] Sep 10 15:56:52.878206 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:52.878069 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:56:52.879012 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:52.878988 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-314824-nl57z"] Sep 10 15:56:52.945531 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:52.945422 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:56:52.949762 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:52.947140 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-314824-55c4c7d84f-szjbr" Sep 10 15:56:53.040935 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:53.040899 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5a17db3-3e2c-421a-bacf-7b9844a8a388-plugins-config-volume\") pod \"e5a17db3-3e2c-421a-bacf-7b9844a8a388\" (UID: \"e5a17db3-3e2c-421a-bacf-7b9844a8a388\") " Sep 10 15:56:53.041123 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:53.040995 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-thjqg\" (UniqueName: \"kubernetes.io/projected/e5a17db3-3e2c-421a-bacf-7b9844a8a388-kube-api-access-thjqg\") pod \"e5a17db3-3e2c-421a-bacf-7b9844a8a388\" (UID: \"e5a17db3-3e2c-421a-bacf-7b9844a8a388\") " Sep 10 15:56:53.041274 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:53.041248 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5a17db3-3e2c-421a-bacf-7b9844a8a388-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e5a17db3-3e2c-421a-bacf-7b9844a8a388" (UID: "e5a17db3-3e2c-421a-bacf-7b9844a8a388"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:56:53.043161 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:53.043141 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5a17db3-3e2c-421a-bacf-7b9844a8a388-kube-api-access-thjqg" (OuterVolumeSpecName: "kube-api-access-thjqg") pod "e5a17db3-3e2c-421a-bacf-7b9844a8a388" (UID: "e5a17db3-3e2c-421a-bacf-7b9844a8a388"). InnerVolumeSpecName "kube-api-access-thjqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:56:53.141858 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:53.141830 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thjqg\" (UniqueName: \"kubernetes.io/projected/e5a17db3-3e2c-421a-bacf-7b9844a8a388-kube-api-access-thjqg\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:56:53.141858 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:53.141856 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5a17db3-3e2c-421a-bacf-7b9844a8a388-plugins-config-volume\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:56:53.169369 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:53.169341 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d833c84-274e-46e6-9920-7b1c8f80f748" path="/var/lib/kubelet/pods/9d833c84-274e-46e6-9920-7b1c8f80f748/volumes" Sep 10 15:56:53.228284 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:53.228224 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:56:53.463121 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:53.463086 2576 generic.go:356] "Generic (PLEG): container finished" podID="e5a17db3-3e2c-421a-bacf-7b9844a8a388" containerID="f9bf73f298e8d429882167b9e56b5b3dfa6937d108e152c2fec14251a597fcd2" exitCode=0 Sep 10 15:56:53.463320 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:53.463139 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-314824-55c4c7d84f-szjbr" Sep 10 15:56:53.463320 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:53.463147 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-314824-55c4c7d84f-szjbr" event={"ID":"e5a17db3-3e2c-421a-bacf-7b9844a8a388","Type":"ContainerDied","Data":"f9bf73f298e8d429882167b9e56b5b3dfa6937d108e152c2fec14251a597fcd2"} Sep 10 15:56:53.463320 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:53.463196 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-314824-55c4c7d84f-szjbr" event={"ID":"e5a17db3-3e2c-421a-bacf-7b9844a8a388","Type":"ContainerDied","Data":"7dd6b3ed0d30d1739aaa43b0a01ae42a35ee6ad801ae223f751e7f8ad4c7d1eb"} Sep 10 15:56:53.463320 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:53.463221 2576 scope.go:117] "RemoveContainer" containerID="f9bf73f298e8d429882167b9e56b5b3dfa6937d108e152c2fec14251a597fcd2" Sep 10 15:56:53.471949 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:53.471926 2576 scope.go:117] "RemoveContainer" containerID="f9bf73f298e8d429882167b9e56b5b3dfa6937d108e152c2fec14251a597fcd2" Sep 10 15:56:53.472184 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:56:53.472164 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9bf73f298e8d429882167b9e56b5b3dfa6937d108e152c2fec14251a597fcd2\": container with ID starting with f9bf73f298e8d429882167b9e56b5b3dfa6937d108e152c2fec14251a597fcd2 not found: ID does not exist" containerID="f9bf73f298e8d429882167b9e56b5b3dfa6937d108e152c2fec14251a597fcd2" Sep 10 15:56:53.472261 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:53.472195 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9bf73f298e8d429882167b9e56b5b3dfa6937d108e152c2fec14251a597fcd2"} err="failed to get container status \"f9bf73f298e8d429882167b9e56b5b3dfa6937d108e152c2fec14251a597fcd2\": rpc error: code = NotFound desc = could not find container \"f9bf73f298e8d429882167b9e56b5b3dfa6937d108e152c2fec14251a597fcd2\": container with ID starting with f9bf73f298e8d429882167b9e56b5b3dfa6937d108e152c2fec14251a597fcd2 not found: ID does not exist" Sep 10 15:56:53.480600 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:53.480580 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-314824-55c4c7d84f-szjbr"] Sep 10 15:56:53.484315 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:53.484297 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-314824-55c4c7d84f-szjbr"] Sep 10 15:56:53.912855 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:53.912817 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-autores-314824-55c4c7d84f-szjbr" podUID="e5a17db3-3e2c-421a-bacf-7b9844a8a388" containerName="payload-processing" probeResult="failure" output="dial tcp 10.133.0.42:9004: i/o timeout" Sep 10 15:56:54.186280 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:54.186200 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 15:56:55.173304 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:55.173270 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5a17db3-3e2c-421a-bacf-7b9844a8a388" path="/var/lib/kubelet/pods/e5a17db3-3e2c-421a-bacf-7b9844a8a388/volumes" Sep 10 15:56:58.314664 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:56:58.314627 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:00.446959 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:00.446924 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-529c3efe-86474f4cd-4zqr6" Sep 10 15:57:04.541175 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:04.541146 2576 generic.go:356] "Generic (PLEG): container finished" podID="780ebc58-fc9c-46d3-b893-6bff1e70a445" containerID="ad2ffb9e031da286e9611cbe501ca978e75463d0e82ed4a79a67c378b6404011" exitCode=0 Sep 10 15:57:04.541586 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:04.541197 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-529c3efe-openshift-default-5b4478f6-595rf" event={"ID":"780ebc58-fc9c-46d3-b893-6bff1e70a445","Type":"ContainerDied","Data":"ad2ffb9e031da286e9611cbe501ca978e75463d0e82ed4a79a67c378b6404011"} Sep 10 15:57:04.541586 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:04.541513 2576 scope.go:117] "RemoveContainer" containerID="ad2ffb9e031da286e9611cbe501ca978e75463d0e82ed4a79a67c378b6404011" Sep 10 15:57:04.542561 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:04.542529 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 15:57:04.542701 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:04.542631 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 15:57:04.542794 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:04.542706 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 15:57:05.547954 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:05.547917 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-529c3efe-openshift-default-5b4478f6-595rf" event={"ID":"780ebc58-fc9c-46d3-b893-6bff1e70a445","Type":"ContainerStarted","Data":"222bdd86bd8d573c59c480aeb878c7ef48dda9416e59d483a07617c94a805b3c"} Sep 10 15:57:06.301522 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:06.301483 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-529c3efe-openshift-default-5b4478f6-595rf" Sep 10 15:57:06.301710 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:06.301599 2576 patch_prober.go:28] interesting pod/e2e-ait-529c3efe-openshift-default-5b4478f6-595rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Sep 10 15:57:06.301710 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:06.301641 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-529c3efe-openshift-default-5b4478f6-595rf" podUID="780ebc58-fc9c-46d3-b893-6bff1e70a445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Sep 10 15:57:07.301030 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:07.300994 2576 patch_prober.go:28] interesting pod/e2e-ait-529c3efe-openshift-default-5b4478f6-595rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Sep 10 15:57:07.301599 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:07.301570 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-529c3efe-openshift-default-5b4478f6-595rf" podUID="780ebc58-fc9c-46d3-b893-6bff1e70a445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Sep 10 15:57:08.300517 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:08.300477 2576 patch_prober.go:28] interesting pod/e2e-ait-529c3efe-openshift-default-5b4478f6-595rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Sep 10 15:57:08.300715 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:08.300557 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-529c3efe-openshift-default-5b4478f6-595rf" podUID="780ebc58-fc9c-46d3-b893-6bff1e70a445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Sep 10 15:57:09.300745 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:09.300689 2576 patch_prober.go:28] interesting pod/e2e-ait-529c3efe-openshift-default-5b4478f6-595rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Sep 10 15:57:09.301249 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:09.300781 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-529c3efe-openshift-default-5b4478f6-595rf" podUID="780ebc58-fc9c-46d3-b893-6bff1e70a445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Sep 10 15:57:09.306612 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:09.306560 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:10.302364 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:10.302320 2576 patch_prober.go:28] interesting pod/e2e-ait-529c3efe-openshift-default-5b4478f6-595rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Sep 10 15:57:10.302879 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:10.302385 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-529c3efe-openshift-default-5b4478f6-595rf" podUID="780ebc58-fc9c-46d3-b893-6bff1e70a445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Sep 10 15:57:10.747809 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:10.747655 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:11.304654 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:11.304624 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-529c3efe-openshift-default-5b4478f6-595rf" Sep 10 15:57:11.305059 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:11.304946 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-529c3efe-openshift-default-5b4478f6-595rf" Sep 10 15:57:11.305517 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:11.305500 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-529c3efe-openshift-default-5b4478f6-595rf" Sep 10 15:57:11.391243 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:11.391213 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 15:57:11.719670 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:11.719587 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-529c3efe-vwphr"] Sep 10 15:57:11.720038 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:11.720025 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5a17db3-3e2c-421a-bacf-7b9844a8a388" containerName="payload-processing" Sep 10 15:57:11.720089 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:11.720040 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5a17db3-3e2c-421a-bacf-7b9844a8a388" containerName="payload-processing" Sep 10 15:57:11.720124 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:11.720119 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5a17db3-3e2c-421a-bacf-7b9844a8a388" containerName="payload-processing" Sep 10 15:57:11.723449 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:11.723431 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-529c3efe-vwphr" Sep 10 15:57:11.726067 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:11.726042 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 15:57:11.726170 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:11.726091 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwwht\"" Sep 10 15:57:11.733318 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:11.733297 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-529c3efe-vwphr"] Sep 10 15:57:11.827353 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:11.827319 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfc217c5-5395-4383-a23c-77dfae1d2699-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-529c3efe-vwphr\" (UID: \"bfc217c5-5395-4383-a23c-77dfae1d2699\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-529c3efe-vwphr" Sep 10 15:57:11.928610 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:11.928561 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfc217c5-5395-4383-a23c-77dfae1d2699-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-529c3efe-vwphr\" (UID: \"bfc217c5-5395-4383-a23c-77dfae1d2699\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-529c3efe-vwphr" Sep 10 15:57:11.929181 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:11.929161 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfc217c5-5395-4383-a23c-77dfae1d2699-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-529c3efe-vwphr\" (UID: \"bfc217c5-5395-4383-a23c-77dfae1d2699\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-529c3efe-vwphr" Sep 10 15:57:12.034614 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:12.034508 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-529c3efe-vwphr" Sep 10 15:57:12.306073 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:12.306048 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-529c3efe-vwphr"] Sep 10 15:57:12.307968 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:57:12.307938 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfc217c5_5395_4383_a23c_77dfae1d2699.slice/crio-9ae5f4370b6165c9df188683c60246f2e29c086702c41590cb7c90ac3b3a5443 WatchSource:0}: Error finding container 9ae5f4370b6165c9df188683c60246f2e29c086702c41590cb7c90ac3b3a5443: Status 404 returned error can't find the container with id 9ae5f4370b6165c9df188683c60246f2e29c086702c41590cb7c90ac3b3a5443 Sep 10 15:57:12.598840 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:12.598726 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-529c3efe-vwphr" event={"ID":"bfc217c5-5395-4383-a23c-77dfae1d2699","Type":"ContainerStarted","Data":"9ae5f4370b6165c9df188683c60246f2e29c086702c41590cb7c90ac3b3a5443"} Sep 10 15:57:13.453350 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:13.453277 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:13.605303 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:13.605268 2576 generic.go:356] "Generic (PLEG): container finished" podID="bfc217c5-5395-4383-a23c-77dfae1d2699" containerID="c609c6570d4f1dbf123ff666b74fc29583da5c6f5789b52073c1867d9f0700fd" exitCode=0 Sep 10 15:57:13.605500 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:13.605358 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-529c3efe-vwphr" event={"ID":"bfc217c5-5395-4383-a23c-77dfae1d2699","Type":"ContainerDied","Data":"c609c6570d4f1dbf123ff666b74fc29583da5c6f5789b52073c1867d9f0700fd"} Sep 10 15:57:14.275485 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:14.275451 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:14.743552 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:14.743531 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-529c3efe-vwphr" Sep 10 15:57:14.858043 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:14.858017 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfc217c5-5395-4383-a23c-77dfae1d2699-maas-api-service-ca\") pod \"bfc217c5-5395-4383-a23c-77dfae1d2699\" (UID: \"bfc217c5-5395-4383-a23c-77dfae1d2699\") " Sep 10 15:57:14.858338 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:14.858314 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfc217c5-5395-4383-a23c-77dfae1d2699-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bfc217c5-5395-4383-a23c-77dfae1d2699" (UID: "bfc217c5-5395-4383-a23c-77dfae1d2699"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:57:14.959220 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:14.959155 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfc217c5-5395-4383-a23c-77dfae1d2699-maas-api-service-ca\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:57:15.284579 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:15.284504 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:15.617173 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:15.617144 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-529c3efe-vwphr" event={"ID":"bfc217c5-5395-4383-a23c-77dfae1d2699","Type":"ContainerDied","Data":"9ae5f4370b6165c9df188683c60246f2e29c086702c41590cb7c90ac3b3a5443"} Sep 10 15:57:15.617347 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:15.617186 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-529c3efe-vwphr" Sep 10 15:57:15.617347 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:15.617195 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ae5f4370b6165c9df188683c60246f2e29c086702c41590cb7c90ac3b3a5443" Sep 10 15:57:17.696844 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:17.696802 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:17.724400 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:17.724370 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-529c3efe-86474f4cd-4zqr6"] Sep 10 15:57:17.724649 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:17.724624 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-529c3efe-86474f4cd-4zqr6" podUID="7143c83a-cf64-40d8-a3f1-8dc1ec11803b" containerName="payload-pre-processing" containerID="cri-o://d3b1d8f7b1f934bb6e20f49718b4a132292fb5cee71a7477407126e18f6809b2" gracePeriod=30 Sep 10 15:57:17.799464 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:17.799414 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:17.897180 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:17.896724 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-529c3efe-vwphr"] Sep 10 15:57:17.898981 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:17.898932 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-529c3efe-vwphr"] Sep 10 15:57:17.901597 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:17.901571 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:17.961849 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:17.961810 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:18.000948 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:17.999902 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-529c3efe-86474f4cd-4zqr6" Sep 10 15:57:18.088692 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:18.088657 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8gqmb\" (UniqueName: \"kubernetes.io/projected/7143c83a-cf64-40d8-a3f1-8dc1ec11803b-kube-api-access-8gqmb\") pod \"7143c83a-cf64-40d8-a3f1-8dc1ec11803b\" (UID: \"7143c83a-cf64-40d8-a3f1-8dc1ec11803b\") " Sep 10 15:57:18.088882 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:18.088710 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7143c83a-cf64-40d8-a3f1-8dc1ec11803b-plugins-config-volume\") pod \"7143c83a-cf64-40d8-a3f1-8dc1ec11803b\" (UID: \"7143c83a-cf64-40d8-a3f1-8dc1ec11803b\") " Sep 10 15:57:18.089085 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:18.089060 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7143c83a-cf64-40d8-a3f1-8dc1ec11803b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7143c83a-cf64-40d8-a3f1-8dc1ec11803b" (UID: "7143c83a-cf64-40d8-a3f1-8dc1ec11803b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:57:18.089191 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:18.089176 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7143c83a-cf64-40d8-a3f1-8dc1ec11803b-plugins-config-volume\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:57:18.090918 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:18.090892 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7143c83a-cf64-40d8-a3f1-8dc1ec11803b-kube-api-access-8gqmb" (OuterVolumeSpecName: "kube-api-access-8gqmb") pod "7143c83a-cf64-40d8-a3f1-8dc1ec11803b" (UID: "7143c83a-cf64-40d8-a3f1-8dc1ec11803b"). InnerVolumeSpecName "kube-api-access-8gqmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:57:18.190621 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:18.190586 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gqmb\" (UniqueName: \"kubernetes.io/projected/7143c83a-cf64-40d8-a3f1-8dc1ec11803b-kube-api-access-8gqmb\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:57:18.634098 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:18.634065 2576 generic.go:356] "Generic (PLEG): container finished" podID="7143c83a-cf64-40d8-a3f1-8dc1ec11803b" containerID="d3b1d8f7b1f934bb6e20f49718b4a132292fb5cee71a7477407126e18f6809b2" exitCode=0 Sep 10 15:57:18.634274 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:18.634110 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-529c3efe-86474f4cd-4zqr6" Sep 10 15:57:18.634274 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:18.634150 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-529c3efe-86474f4cd-4zqr6" event={"ID":"7143c83a-cf64-40d8-a3f1-8dc1ec11803b","Type":"ContainerDied","Data":"d3b1d8f7b1f934bb6e20f49718b4a132292fb5cee71a7477407126e18f6809b2"} Sep 10 15:57:18.634274 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:18.634210 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-529c3efe-86474f4cd-4zqr6" event={"ID":"7143c83a-cf64-40d8-a3f1-8dc1ec11803b","Type":"ContainerDied","Data":"0eae1771126a54989a8e4a6d2385d1591697675711728b7dcdc1f94a39773e4e"} Sep 10 15:57:18.634274 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:18.634232 2576 scope.go:117] "RemoveContainer" containerID="d3b1d8f7b1f934bb6e20f49718b4a132292fb5cee71a7477407126e18f6809b2" Sep 10 15:57:18.643184 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:18.643166 2576 scope.go:117] "RemoveContainer" containerID="d3b1d8f7b1f934bb6e20f49718b4a132292fb5cee71a7477407126e18f6809b2" Sep 10 15:57:18.643454 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:57:18.643435 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3b1d8f7b1f934bb6e20f49718b4a132292fb5cee71a7477407126e18f6809b2\": container with ID starting with d3b1d8f7b1f934bb6e20f49718b4a132292fb5cee71a7477407126e18f6809b2 not found: ID does not exist" containerID="d3b1d8f7b1f934bb6e20f49718b4a132292fb5cee71a7477407126e18f6809b2" Sep 10 15:57:18.643503 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:18.643464 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3b1d8f7b1f934bb6e20f49718b4a132292fb5cee71a7477407126e18f6809b2"} err="failed to get container status \"d3b1d8f7b1f934bb6e20f49718b4a132292fb5cee71a7477407126e18f6809b2\": rpc error: code = NotFound desc = could not find container \"d3b1d8f7b1f934bb6e20f49718b4a132292fb5cee71a7477407126e18f6809b2\": container with ID starting with d3b1d8f7b1f934bb6e20f49718b4a132292fb5cee71a7477407126e18f6809b2 not found: ID does not exist" Sep 10 15:57:18.658315 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:18.658294 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-529c3efe-86474f4cd-4zqr6"] Sep 10 15:57:18.666070 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:18.666051 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-529c3efe-86474f4cd-4zqr6"] Sep 10 15:57:19.027753 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:19.027652 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-529c3efe-openshift-default-5b4478f6-595rf"] Sep 10 15:57:19.028105 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:19.027879 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-529c3efe-openshift-default-5b4478f6-595rf" podUID="780ebc58-fc9c-46d3-b893-6bff1e70a445" containerName="istio-proxy" containerID="cri-o://222bdd86bd8d573c59c480aeb878c7ef48dda9416e59d483a07617c94a805b3c" gracePeriod=30 Sep 10 15:57:19.052114 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:19.052074 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:19.169315 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:19.169285 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7143c83a-cf64-40d8-a3f1-8dc1ec11803b" path="/var/lib/kubelet/pods/7143c83a-cf64-40d8-a3f1-8dc1ec11803b/volumes" Sep 10 15:57:19.169654 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:19.169640 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfc217c5-5395-4383-a23c-77dfae1d2699" path="/var/lib/kubelet/pods/bfc217c5-5395-4383-a23c-77dfae1d2699/volumes" Sep 10 15:57:19.467554 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:19.467524 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:19.818627 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:19.818587 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2"] Sep 10 15:57:19.819080 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:19.819062 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfc217c5-5395-4383-a23c-77dfae1d2699" containerName="revoke-keys" Sep 10 15:57:19.819193 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:19.819082 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfc217c5-5395-4383-a23c-77dfae1d2699" containerName="revoke-keys" Sep 10 15:57:19.819193 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:19.819097 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7143c83a-cf64-40d8-a3f1-8dc1ec11803b" containerName="payload-pre-processing" Sep 10 15:57:19.819193 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:19.819104 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7143c83a-cf64-40d8-a3f1-8dc1ec11803b" containerName="payload-pre-processing" Sep 10 15:57:19.819339 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:19.819214 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfc217c5-5395-4383-a23c-77dfae1d2699" containerName="revoke-keys" Sep 10 15:57:19.819339 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:19.819227 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7143c83a-cf64-40d8-a3f1-8dc1ec11803b" containerName="payload-pre-processing" Sep 10 15:57:19.823613 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:19.823592 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2" Sep 10 15:57:19.828170 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:19.828149 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-7a3187a1-openshift-default-dockercfg-s2t4f\"" Sep 10 15:57:19.840032 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:19.840008 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2"] Sep 10 15:57:19.846292 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:19.846267 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:19.904416 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:19.904386 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2af68ba0-44eb-40fc-b104-ead9c83dcd17-workload-certs\") pod \"e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2\" (UID: \"2af68ba0-44eb-40fc-b104-ead9c83dcd17\") " pod="openshift-ingress/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2" Sep 10 15:57:19.904612 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:19.904596 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2af68ba0-44eb-40fc-b104-ead9c83dcd17-istio-data\") pod \"e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2\" (UID: \"2af68ba0-44eb-40fc-b104-ead9c83dcd17\") " pod="openshift-ingress/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2" Sep 10 15:57:19.904809 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:19.904782 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2af68ba0-44eb-40fc-b104-ead9c83dcd17-workload-socket\") pod \"e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2\" (UID: \"2af68ba0-44eb-40fc-b104-ead9c83dcd17\") " pod="openshift-ingress/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2" Sep 10 15:57:19.904930 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:19.904817 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2af68ba0-44eb-40fc-b104-ead9c83dcd17-istiod-ca-cert\") pod \"e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2\" (UID: \"2af68ba0-44eb-40fc-b104-ead9c83dcd17\") " pod="openshift-ingress/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2" Sep 10 15:57:19.904930 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:19.904852 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2af68ba0-44eb-40fc-b104-ead9c83dcd17-istio-token\") pod \"e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2\" (UID: \"2af68ba0-44eb-40fc-b104-ead9c83dcd17\") " pod="openshift-ingress/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2" Sep 10 15:57:19.905040 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:19.904961 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs6gb\" (UniqueName: \"kubernetes.io/projected/2af68ba0-44eb-40fc-b104-ead9c83dcd17-kube-api-access-rs6gb\") pod \"e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2\" (UID: \"2af68ba0-44eb-40fc-b104-ead9c83dcd17\") " pod="openshift-ingress/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2" Sep 10 15:57:19.905095 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:19.905043 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2af68ba0-44eb-40fc-b104-ead9c83dcd17-istio-podinfo\") pod \"e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2\" (UID: \"2af68ba0-44eb-40fc-b104-ead9c83dcd17\") " pod="openshift-ingress/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2" Sep 10 15:57:19.905149 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:19.905103 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2af68ba0-44eb-40fc-b104-ead9c83dcd17-istio-envoy\") pod \"e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2\" (UID: \"2af68ba0-44eb-40fc-b104-ead9c83dcd17\") " pod="openshift-ingress/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2" Sep 10 15:57:19.905203 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:19.905176 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2af68ba0-44eb-40fc-b104-ead9c83dcd17-credential-socket\") pod \"e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2\" (UID: \"2af68ba0-44eb-40fc-b104-ead9c83dcd17\") " pod="openshift-ingress/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2" Sep 10 15:57:20.006263 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:20.006222 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2af68ba0-44eb-40fc-b104-ead9c83dcd17-istio-envoy\") pod \"e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2\" (UID: \"2af68ba0-44eb-40fc-b104-ead9c83dcd17\") " pod="openshift-ingress/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2" Sep 10 15:57:20.006468 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:20.006313 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2af68ba0-44eb-40fc-b104-ead9c83dcd17-credential-socket\") pod \"e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2\" (UID: \"2af68ba0-44eb-40fc-b104-ead9c83dcd17\") " pod="openshift-ingress/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2" Sep 10 15:57:20.006468 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:20.006363 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2af68ba0-44eb-40fc-b104-ead9c83dcd17-workload-certs\") pod \"e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2\" (UID: \"2af68ba0-44eb-40fc-b104-ead9c83dcd17\") " pod="openshift-ingress/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2" Sep 10 15:57:20.006468 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:20.006400 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2af68ba0-44eb-40fc-b104-ead9c83dcd17-istio-data\") pod \"e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2\" (UID: \"2af68ba0-44eb-40fc-b104-ead9c83dcd17\") " pod="openshift-ingress/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2" Sep 10 15:57:20.006468 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:20.006465 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2af68ba0-44eb-40fc-b104-ead9c83dcd17-workload-socket\") pod \"e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2\" (UID: \"2af68ba0-44eb-40fc-b104-ead9c83dcd17\") " pod="openshift-ingress/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2" Sep 10 15:57:20.006765 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:20.006493 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2af68ba0-44eb-40fc-b104-ead9c83dcd17-istiod-ca-cert\") pod \"e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2\" (UID: \"2af68ba0-44eb-40fc-b104-ead9c83dcd17\") " pod="openshift-ingress/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2" Sep 10 15:57:20.006765 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:20.006530 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2af68ba0-44eb-40fc-b104-ead9c83dcd17-istio-token\") pod \"e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2\" (UID: \"2af68ba0-44eb-40fc-b104-ead9c83dcd17\") " pod="openshift-ingress/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2" Sep 10 15:57:20.006765 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:20.006587 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs6gb\" (UniqueName: \"kubernetes.io/projected/2af68ba0-44eb-40fc-b104-ead9c83dcd17-kube-api-access-rs6gb\") pod \"e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2\" (UID: \"2af68ba0-44eb-40fc-b104-ead9c83dcd17\") " pod="openshift-ingress/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2" Sep 10 15:57:20.006765 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:20.006619 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2af68ba0-44eb-40fc-b104-ead9c83dcd17-istio-podinfo\") pod \"e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2\" (UID: \"2af68ba0-44eb-40fc-b104-ead9c83dcd17\") " pod="openshift-ingress/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2" Sep 10 15:57:20.006765 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:20.006706 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2af68ba0-44eb-40fc-b104-ead9c83dcd17-credential-socket\") pod \"e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2\" (UID: \"2af68ba0-44eb-40fc-b104-ead9c83dcd17\") " pod="openshift-ingress/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2" Sep 10 15:57:20.007057 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:20.006899 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2af68ba0-44eb-40fc-b104-ead9c83dcd17-workload-socket\") pod \"e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2\" (UID: \"2af68ba0-44eb-40fc-b104-ead9c83dcd17\") " pod="openshift-ingress/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2" Sep 10 15:57:20.007129 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:20.007109 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2af68ba0-44eb-40fc-b104-ead9c83dcd17-workload-certs\") pod \"e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2\" (UID: \"2af68ba0-44eb-40fc-b104-ead9c83dcd17\") " pod="openshift-ingress/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2" Sep 10 15:57:20.007364 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:20.007342 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2af68ba0-44eb-40fc-b104-ead9c83dcd17-istio-data\") pod \"e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2\" (UID: \"2af68ba0-44eb-40fc-b104-ead9c83dcd17\") " pod="openshift-ingress/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2" Sep 10 15:57:20.007833 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:20.007752 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2af68ba0-44eb-40fc-b104-ead9c83dcd17-istiod-ca-cert\") pod \"e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2\" (UID: \"2af68ba0-44eb-40fc-b104-ead9c83dcd17\") " pod="openshift-ingress/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2" Sep 10 15:57:20.009811 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:20.009787 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2af68ba0-44eb-40fc-b104-ead9c83dcd17-istio-envoy\") pod \"e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2\" (UID: \"2af68ba0-44eb-40fc-b104-ead9c83dcd17\") " pod="openshift-ingress/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2" Sep 10 15:57:20.010249 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:20.010221 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2af68ba0-44eb-40fc-b104-ead9c83dcd17-istio-podinfo\") pod \"e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2\" (UID: \"2af68ba0-44eb-40fc-b104-ead9c83dcd17\") " pod="openshift-ingress/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2" Sep 10 15:57:20.014890 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:20.014867 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2af68ba0-44eb-40fc-b104-ead9c83dcd17-istio-token\") pod \"e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2\" (UID: \"2af68ba0-44eb-40fc-b104-ead9c83dcd17\") " pod="openshift-ingress/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2" Sep 10 15:57:20.015431 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:20.015409 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs6gb\" (UniqueName: \"kubernetes.io/projected/2af68ba0-44eb-40fc-b104-ead9c83dcd17-kube-api-access-rs6gb\") pod \"e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2\" (UID: \"2af68ba0-44eb-40fc-b104-ead9c83dcd17\") " pod="openshift-ingress/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2" Sep 10 15:57:20.115593 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:20.115496 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:20.134098 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:20.134061 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2" Sep 10 15:57:20.423595 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:20.423517 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2"] Sep 10 15:57:20.424904 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:57:20.424880 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2af68ba0_44eb_40fc_b104_ead9c83dcd17.slice/crio-e3d85ca975b64ba6651fd889985c80f267f6101f6606c60d4d085d0b92aa9018 WatchSource:0}: Error finding container e3d85ca975b64ba6651fd889985c80f267f6101f6606c60d4d085d0b92aa9018: Status 404 returned error can't find the container with id e3d85ca975b64ba6651fd889985c80f267f6101f6606c60d4d085d0b92aa9018 Sep 10 15:57:20.427086 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:20.427055 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 15:57:20.427178 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:20.427120 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 15:57:20.427178 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:20.427150 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 15:57:20.648296 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:20.648258 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2" event={"ID":"2af68ba0-44eb-40fc-b104-ead9c83dcd17","Type":"ContainerStarted","Data":"9a29c5a8623d1137d00fcb1de40e828c7d0ec6045a3bae5a5fb774cd7b4d63ef"} Sep 10 15:57:20.648476 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:20.648321 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2" event={"ID":"2af68ba0-44eb-40fc-b104-ead9c83dcd17","Type":"ContainerStarted","Data":"e3d85ca975b64ba6651fd889985c80f267f6101f6606c60d4d085d0b92aa9018"} Sep 10 15:57:20.671884 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:20.671776 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2" podStartSLOduration=1.671753928 podStartE2EDuration="1.671753928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:57:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:57:20.668334928 +0000 UTC m=+918.021854112" watchObservedRunningTime="2026-09-10 15:57:20.671753928 +0000 UTC m=+918.025273087" Sep 10 15:57:21.135479 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:21.135446 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2" Sep 10 15:57:21.140403 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:21.140377 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2" Sep 10 15:57:21.653423 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:21.653388 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2" Sep 10 15:57:21.654364 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:21.654344 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2" Sep 10 15:57:22.006862 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:22.006756 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:22.100132 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:22.100093 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:23.176329 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:23.176291 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:23.279768 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:23.279704 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:23.377287 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:23.377254 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:23.836686 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:23.836650 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:23.954435 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:23.954404 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:24.110724 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:24.110642 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-7a3187a1-6cdc7b6bbf-vv998"] Sep 10 15:57:24.114666 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:24.114641 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-7a3187a1-6cdc7b6bbf-vv998" Sep 10 15:57:24.117382 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:24.117359 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-7a3187a1\"" Sep 10 15:57:24.117839 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:24.117821 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-7a3187a1-dockercfg-57wtg\"" Sep 10 15:57:24.128280 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:24.128255 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-7a3187a1-6cdc7b6bbf-vv998"] Sep 10 15:57:24.250404 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:24.250370 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnvpc\" (UniqueName: \"kubernetes.io/projected/d6afd816-e72c-414e-a395-37f788ef3a43-kube-api-access-dnvpc\") pod \"payload-processing-e2e-derive-7a3187a1-6cdc7b6bbf-vv998\" (UID: \"d6afd816-e72c-414e-a395-37f788ef3a43\") " pod="openshift-ingress/payload-processing-e2e-derive-7a3187a1-6cdc7b6bbf-vv998" Sep 10 15:57:24.250793 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:24.250560 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6afd816-e72c-414e-a395-37f788ef3a43-plugins-config-volume\") pod \"payload-processing-e2e-derive-7a3187a1-6cdc7b6bbf-vv998\" (UID: \"d6afd816-e72c-414e-a395-37f788ef3a43\") " pod="openshift-ingress/payload-processing-e2e-derive-7a3187a1-6cdc7b6bbf-vv998" Sep 10 15:57:24.287511 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:24.287487 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-529c3efe-openshift-default-5b4478f6-595rf" Sep 10 15:57:24.351226 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:24.351198 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/780ebc58-fc9c-46d3-b893-6bff1e70a445-istio-podinfo\") pod \"780ebc58-fc9c-46d3-b893-6bff1e70a445\" (UID: \"780ebc58-fc9c-46d3-b893-6bff1e70a445\") " Sep 10 15:57:24.351368 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:24.351238 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/780ebc58-fc9c-46d3-b893-6bff1e70a445-istio-token\") pod \"780ebc58-fc9c-46d3-b893-6bff1e70a445\" (UID: \"780ebc58-fc9c-46d3-b893-6bff1e70a445\") " Sep 10 15:57:24.351368 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:24.351255 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/780ebc58-fc9c-46d3-b893-6bff1e70a445-workload-socket\") pod \"780ebc58-fc9c-46d3-b893-6bff1e70a445\" (UID: \"780ebc58-fc9c-46d3-b893-6bff1e70a445\") " Sep 10 15:57:24.351368 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:24.351283 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/780ebc58-fc9c-46d3-b893-6bff1e70a445-credential-socket\") pod \"780ebc58-fc9c-46d3-b893-6bff1e70a445\" (UID: \"780ebc58-fc9c-46d3-b893-6bff1e70a445\") " Sep 10 15:57:24.351536 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:24.351409 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/780ebc58-fc9c-46d3-b893-6bff1e70a445-workload-certs\") pod \"780ebc58-fc9c-46d3-b893-6bff1e70a445\" (UID: \"780ebc58-fc9c-46d3-b893-6bff1e70a445\") " Sep 10 15:57:24.351536 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:24.351441 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/780ebc58-fc9c-46d3-b893-6bff1e70a445-istio-data\") pod \"780ebc58-fc9c-46d3-b893-6bff1e70a445\" (UID: \"780ebc58-fc9c-46d3-b893-6bff1e70a445\") " Sep 10 15:57:24.351536 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:24.351519 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pq6b4\" (UniqueName: \"kubernetes.io/projected/780ebc58-fc9c-46d3-b893-6bff1e70a445-kube-api-access-pq6b4\") pod \"780ebc58-fc9c-46d3-b893-6bff1e70a445\" (UID: \"780ebc58-fc9c-46d3-b893-6bff1e70a445\") " Sep 10 15:57:24.351536 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:24.351521 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/780ebc58-fc9c-46d3-b893-6bff1e70a445-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "780ebc58-fc9c-46d3-b893-6bff1e70a445" (UID: "780ebc58-fc9c-46d3-b893-6bff1e70a445"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:57:24.351782 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:24.351544 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/780ebc58-fc9c-46d3-b893-6bff1e70a445-istiod-ca-cert\") pod \"780ebc58-fc9c-46d3-b893-6bff1e70a445\" (UID: \"780ebc58-fc9c-46d3-b893-6bff1e70a445\") " Sep 10 15:57:24.351782 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:24.351617 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/780ebc58-fc9c-46d3-b893-6bff1e70a445-istio-envoy\") pod \"780ebc58-fc9c-46d3-b893-6bff1e70a445\" (UID: \"780ebc58-fc9c-46d3-b893-6bff1e70a445\") " Sep 10 15:57:24.351782 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:24.351604 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/780ebc58-fc9c-46d3-b893-6bff1e70a445-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "780ebc58-fc9c-46d3-b893-6bff1e70a445" (UID: "780ebc58-fc9c-46d3-b893-6bff1e70a445"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:57:24.351782 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:24.351675 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/780ebc58-fc9c-46d3-b893-6bff1e70a445-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "780ebc58-fc9c-46d3-b893-6bff1e70a445" (UID: "780ebc58-fc9c-46d3-b893-6bff1e70a445"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:57:24.351782 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:24.351765 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/780ebc58-fc9c-46d3-b893-6bff1e70a445-istio-data" (OuterVolumeSpecName: "istio-data") pod "780ebc58-fc9c-46d3-b893-6bff1e70a445" (UID: "780ebc58-fc9c-46d3-b893-6bff1e70a445"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:57:24.352196 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:24.351921 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/780ebc58-fc9c-46d3-b893-6bff1e70a445-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "780ebc58-fc9c-46d3-b893-6bff1e70a445" (UID: "780ebc58-fc9c-46d3-b893-6bff1e70a445"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:57:24.352196 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:24.351929 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6afd816-e72c-414e-a395-37f788ef3a43-plugins-config-volume\") pod \"payload-processing-e2e-derive-7a3187a1-6cdc7b6bbf-vv998\" (UID: \"d6afd816-e72c-414e-a395-37f788ef3a43\") " pod="openshift-ingress/payload-processing-e2e-derive-7a3187a1-6cdc7b6bbf-vv998" Sep 10 15:57:24.352196 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:24.351983 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnvpc\" (UniqueName: \"kubernetes.io/projected/d6afd816-e72c-414e-a395-37f788ef3a43-kube-api-access-dnvpc\") pod \"payload-processing-e2e-derive-7a3187a1-6cdc7b6bbf-vv998\" (UID: \"d6afd816-e72c-414e-a395-37f788ef3a43\") " pod="openshift-ingress/payload-processing-e2e-derive-7a3187a1-6cdc7b6bbf-vv998" Sep 10 15:57:24.352196 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:24.352102 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/780ebc58-fc9c-46d3-b893-6bff1e70a445-workload-socket\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:57:24.352196 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:24.352117 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/780ebc58-fc9c-46d3-b893-6bff1e70a445-credential-socket\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:57:24.352196 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:24.352129 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/780ebc58-fc9c-46d3-b893-6bff1e70a445-workload-certs\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:57:24.352196 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:24.352142 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/780ebc58-fc9c-46d3-b893-6bff1e70a445-istio-data\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:57:24.352196 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:24.352153 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/780ebc58-fc9c-46d3-b893-6bff1e70a445-istiod-ca-cert\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:57:24.352989 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:24.352961 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6afd816-e72c-414e-a395-37f788ef3a43-plugins-config-volume\") pod \"payload-processing-e2e-derive-7a3187a1-6cdc7b6bbf-vv998\" (UID: \"d6afd816-e72c-414e-a395-37f788ef3a43\") " pod="openshift-ingress/payload-processing-e2e-derive-7a3187a1-6cdc7b6bbf-vv998" Sep 10 15:57:24.353771 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:24.353718 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/780ebc58-fc9c-46d3-b893-6bff1e70a445-istio-token" (OuterVolumeSpecName: "istio-token") pod "780ebc58-fc9c-46d3-b893-6bff1e70a445" (UID: "780ebc58-fc9c-46d3-b893-6bff1e70a445"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:57:24.353923 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:24.353907 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/780ebc58-fc9c-46d3-b893-6bff1e70a445-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "780ebc58-fc9c-46d3-b893-6bff1e70a445" (UID: "780ebc58-fc9c-46d3-b893-6bff1e70a445"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:57:24.354125 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:24.354109 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/780ebc58-fc9c-46d3-b893-6bff1e70a445-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "780ebc58-fc9c-46d3-b893-6bff1e70a445" (UID: "780ebc58-fc9c-46d3-b893-6bff1e70a445"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 15:57:24.354186 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:24.354114 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/780ebc58-fc9c-46d3-b893-6bff1e70a445-kube-api-access-pq6b4" (OuterVolumeSpecName: "kube-api-access-pq6b4") pod "780ebc58-fc9c-46d3-b893-6bff1e70a445" (UID: "780ebc58-fc9c-46d3-b893-6bff1e70a445"). InnerVolumeSpecName "kube-api-access-pq6b4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:57:24.359874 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:24.359856 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnvpc\" (UniqueName: \"kubernetes.io/projected/d6afd816-e72c-414e-a395-37f788ef3a43-kube-api-access-dnvpc\") pod \"payload-processing-e2e-derive-7a3187a1-6cdc7b6bbf-vv998\" (UID: \"d6afd816-e72c-414e-a395-37f788ef3a43\") " pod="openshift-ingress/payload-processing-e2e-derive-7a3187a1-6cdc7b6bbf-vv998" Sep 10 15:57:24.429305 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:24.429245 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-7a3187a1-6cdc7b6bbf-vv998" Sep 10 15:57:24.452942 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:24.452920 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/780ebc58-fc9c-46d3-b893-6bff1e70a445-istio-podinfo\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:57:24.452942 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:24.452943 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/780ebc58-fc9c-46d3-b893-6bff1e70a445-istio-token\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:57:24.453077 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:24.452953 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pq6b4\" (UniqueName: \"kubernetes.io/projected/780ebc58-fc9c-46d3-b893-6bff1e70a445-kube-api-access-pq6b4\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:57:24.453077 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:24.452962 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/780ebc58-fc9c-46d3-b893-6bff1e70a445-istio-envoy\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:57:24.669571 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:24.669538 2576 generic.go:356] "Generic (PLEG): container finished" podID="780ebc58-fc9c-46d3-b893-6bff1e70a445" containerID="222bdd86bd8d573c59c480aeb878c7ef48dda9416e59d483a07617c94a805b3c" exitCode=0 Sep 10 15:57:24.669717 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:24.669587 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-529c3efe-openshift-default-5b4478f6-595rf" Sep 10 15:57:24.669717 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:24.669616 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-529c3efe-openshift-default-5b4478f6-595rf" event={"ID":"780ebc58-fc9c-46d3-b893-6bff1e70a445","Type":"ContainerDied","Data":"222bdd86bd8d573c59c480aeb878c7ef48dda9416e59d483a07617c94a805b3c"} Sep 10 15:57:24.669717 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:24.669658 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-529c3efe-openshift-default-5b4478f6-595rf" event={"ID":"780ebc58-fc9c-46d3-b893-6bff1e70a445","Type":"ContainerDied","Data":"cdffbb030f17767f6e0f23a24796dd580191e2177880d1136bb5a9c7d584c0df"} Sep 10 15:57:24.669717 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:24.669679 2576 scope.go:117] "RemoveContainer" containerID="222bdd86bd8d573c59c480aeb878c7ef48dda9416e59d483a07617c94a805b3c" Sep 10 15:57:24.678481 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:24.678464 2576 scope.go:117] "RemoveContainer" containerID="ad2ffb9e031da286e9611cbe501ca978e75463d0e82ed4a79a67c378b6404011" Sep 10 15:57:24.686502 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:24.686482 2576 scope.go:117] "RemoveContainer" containerID="222bdd86bd8d573c59c480aeb878c7ef48dda9416e59d483a07617c94a805b3c" Sep 10 15:57:24.686792 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:57:24.686770 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"222bdd86bd8d573c59c480aeb878c7ef48dda9416e59d483a07617c94a805b3c\": container with ID starting with 222bdd86bd8d573c59c480aeb878c7ef48dda9416e59d483a07617c94a805b3c not found: ID does not exist" containerID="222bdd86bd8d573c59c480aeb878c7ef48dda9416e59d483a07617c94a805b3c" Sep 10 15:57:24.686855 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:24.686800 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"222bdd86bd8d573c59c480aeb878c7ef48dda9416e59d483a07617c94a805b3c"} err="failed to get container status \"222bdd86bd8d573c59c480aeb878c7ef48dda9416e59d483a07617c94a805b3c\": rpc error: code = NotFound desc = could not find container \"222bdd86bd8d573c59c480aeb878c7ef48dda9416e59d483a07617c94a805b3c\": container with ID starting with 222bdd86bd8d573c59c480aeb878c7ef48dda9416e59d483a07617c94a805b3c not found: ID does not exist" Sep 10 15:57:24.686855 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:24.686815 2576 scope.go:117] "RemoveContainer" containerID="ad2ffb9e031da286e9611cbe501ca978e75463d0e82ed4a79a67c378b6404011" Sep 10 15:57:24.687049 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:57:24.687033 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad2ffb9e031da286e9611cbe501ca978e75463d0e82ed4a79a67c378b6404011\": container with ID starting with ad2ffb9e031da286e9611cbe501ca978e75463d0e82ed4a79a67c378b6404011 not found: ID does not exist" containerID="ad2ffb9e031da286e9611cbe501ca978e75463d0e82ed4a79a67c378b6404011" Sep 10 15:57:24.687098 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:24.687053 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad2ffb9e031da286e9611cbe501ca978e75463d0e82ed4a79a67c378b6404011"} err="failed to get container status \"ad2ffb9e031da286e9611cbe501ca978e75463d0e82ed4a79a67c378b6404011\": rpc error: code = NotFound desc = could not find container \"ad2ffb9e031da286e9611cbe501ca978e75463d0e82ed4a79a67c378b6404011\": container with ID starting with ad2ffb9e031da286e9611cbe501ca978e75463d0e82ed4a79a67c378b6404011 not found: ID does not exist" Sep 10 15:57:24.692155 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:24.692134 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-529c3efe-openshift-default-5b4478f6-595rf"] Sep 10 15:57:24.698201 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:24.698178 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-529c3efe-openshift-default-5b4478f6-595rf"] Sep 10 15:57:24.701297 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:24.701281 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-7a3187a1-6cdc7b6bbf-vv998"] Sep 10 15:57:24.703347 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:57:24.703324 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6afd816_e72c_414e_a395_37f788ef3a43.slice/crio-1e490a50fdc0c004aaf9f7d6c7283adffc1448035518a4401643d974652f03e4 WatchSource:0}: Error finding container 1e490a50fdc0c004aaf9f7d6c7283adffc1448035518a4401643d974652f03e4: Status 404 returned error can't find the container with id 1e490a50fdc0c004aaf9f7d6c7283adffc1448035518a4401643d974652f03e4 Sep 10 15:57:25.178723 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:25.178680 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="780ebc58-fc9c-46d3-b893-6bff1e70a445" path="/var/lib/kubelet/pods/780ebc58-fc9c-46d3-b893-6bff1e70a445/volumes" Sep 10 15:57:25.676648 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:25.676616 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-7a3187a1-6cdc7b6bbf-vv998" event={"ID":"d6afd816-e72c-414e-a395-37f788ef3a43","Type":"ContainerStarted","Data":"d70b7c95116ee8aa05a269b8cf6b77afdc739b99cc072bd1d734f7253df42e88"} Sep 10 15:57:25.677092 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:25.676666 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-7a3187a1-6cdc7b6bbf-vv998" event={"ID":"d6afd816-e72c-414e-a395-37f788ef3a43","Type":"ContainerStarted","Data":"1e490a50fdc0c004aaf9f7d6c7283adffc1448035518a4401643d974652f03e4"} Sep 10 15:57:25.677092 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:25.676694 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-7a3187a1-6cdc7b6bbf-vv998" Sep 10 15:57:25.696936 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:25.696874 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-7a3187a1-6cdc7b6bbf-vv998" podStartSLOduration=1.6968573999999998 podStartE2EDuration="1.6968574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:57:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:57:25.694298296 +0000 UTC m=+923.047817479" watchObservedRunningTime="2026-09-10 15:57:25.6968574 +0000 UTC m=+923.050376584" Sep 10 15:57:27.686992 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:27.686961 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 15:57:35.161318 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:35.161249 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:35.396534 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:35.396496 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:35.859500 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:35.859459 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:36.216947 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:36.216851 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:36.684275 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:36.684247 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-7a3187a1-6cdc7b6bbf-vv998" Sep 10 15:57:36.910364 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:36.910327 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:37.271883 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:37.270661 2576 patch_prober.go:28] interesting pod/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": read tcp 10.133.0.2:43008->10.133.0.50:15021: read: connection reset by peer" start-of-body= Sep 10 15:57:37.271883 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:37.270725 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2" podUID="2af68ba0-44eb-40fc-b104-ead9c83dcd17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": read tcp 10.133.0.2:43008->10.133.0.50:15021: read: connection reset by peer" Sep 10 15:57:37.756806 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:37.756769 2576 generic.go:356] "Generic (PLEG): container finished" podID="2af68ba0-44eb-40fc-b104-ead9c83dcd17" containerID="9a29c5a8623d1137d00fcb1de40e828c7d0ec6045a3bae5a5fb774cd7b4d63ef" exitCode=0 Sep 10 15:57:37.757000 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:37.756825 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2" event={"ID":"2af68ba0-44eb-40fc-b104-ead9c83dcd17","Type":"ContainerDied","Data":"9a29c5a8623d1137d00fcb1de40e828c7d0ec6045a3bae5a5fb774cd7b4d63ef"} Sep 10 15:57:37.757247 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:37.757229 2576 scope.go:117] "RemoveContainer" containerID="9a29c5a8623d1137d00fcb1de40e828c7d0ec6045a3bae5a5fb774cd7b4d63ef" Sep 10 15:57:37.758576 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:37.758532 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 15:57:37.758683 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:37.758662 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 15:57:37.758767 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:37.758719 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 15:57:38.771758 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:38.771432 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2" event={"ID":"2af68ba0-44eb-40fc-b104-ead9c83dcd17","Type":"ContainerStarted","Data":"c415822991484e3bfcd696e6878f6184aa0fccca3d551b567181210b80c1ac61"} Sep 10 15:57:39.134758 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:39.134703 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2" Sep 10 15:57:39.135013 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:39.134987 2576 patch_prober.go:28] interesting pod/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Sep 10 15:57:39.135108 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:39.135045 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2" podUID="2af68ba0-44eb-40fc-b104-ead9c83dcd17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Sep 10 15:57:39.717404 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:39.717366 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:39.835472 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:39.835437 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:40.135223 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:40.135189 2576 patch_prober.go:28] interesting pod/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Sep 10 15:57:40.135393 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:40.135244 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2" podUID="2af68ba0-44eb-40fc-b104-ead9c83dcd17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Sep 10 15:57:40.908756 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:40.901215 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:40.996327 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:40.996294 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:41.135369 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:41.135338 2576 patch_prober.go:28] interesting pod/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Sep 10 15:57:41.135539 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:41.135395 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2" podUID="2af68ba0-44eb-40fc-b104-ead9c83dcd17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Sep 10 15:57:41.543574 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:41.543534 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:41.667253 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:41.667219 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:41.851560 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:41.851484 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-5c1d26-77c58dd5df-lfjmn"] Sep 10 15:57:41.851957 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:41.851943 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="780ebc58-fc9c-46d3-b893-6bff1e70a445" containerName="istio-proxy" Sep 10 15:57:41.852008 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:41.851959 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="780ebc58-fc9c-46d3-b893-6bff1e70a445" containerName="istio-proxy" Sep 10 15:57:41.852044 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:41.852035 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="780ebc58-fc9c-46d3-b893-6bff1e70a445" containerName="istio-proxy" Sep 10 15:57:41.852081 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:41.852044 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="780ebc58-fc9c-46d3-b893-6bff1e70a445" containerName="istio-proxy" Sep 10 15:57:41.855252 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:41.855237 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-5c1d26-77c58dd5df-lfjmn" Sep 10 15:57:41.857676 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:41.857653 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-5c1d26\"" Sep 10 15:57:41.857854 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:41.857837 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-5c1d26-dockercfg-hvws8\"" Sep 10 15:57:41.872978 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:41.872955 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-5c1d26-77c58dd5df-lfjmn"] Sep 10 15:57:41.939081 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:41.939045 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae00a62c-0c5b-4e28-ac4b-bfc19709dd99-plugins-config-volume\") pod \"payload-processing-e2e-explicit-5c1d26-77c58dd5df-lfjmn\" (UID: \"ae00a62c-0c5b-4e28-ac4b-bfc19709dd99\") " pod="openshift-ingress/payload-processing-e2e-explicit-5c1d26-77c58dd5df-lfjmn" Sep 10 15:57:41.939441 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:41.939088 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbrtr\" (UniqueName: \"kubernetes.io/projected/ae00a62c-0c5b-4e28-ac4b-bfc19709dd99-kube-api-access-lbrtr\") pod \"payload-processing-e2e-explicit-5c1d26-77c58dd5df-lfjmn\" (UID: \"ae00a62c-0c5b-4e28-ac4b-bfc19709dd99\") " pod="openshift-ingress/payload-processing-e2e-explicit-5c1d26-77c58dd5df-lfjmn" Sep 10 15:57:42.039681 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:42.039643 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae00a62c-0c5b-4e28-ac4b-bfc19709dd99-plugins-config-volume\") pod \"payload-processing-e2e-explicit-5c1d26-77c58dd5df-lfjmn\" (UID: \"ae00a62c-0c5b-4e28-ac4b-bfc19709dd99\") " pod="openshift-ingress/payload-processing-e2e-explicit-5c1d26-77c58dd5df-lfjmn" Sep 10 15:57:42.039877 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:42.039691 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbrtr\" (UniqueName: \"kubernetes.io/projected/ae00a62c-0c5b-4e28-ac4b-bfc19709dd99-kube-api-access-lbrtr\") pod \"payload-processing-e2e-explicit-5c1d26-77c58dd5df-lfjmn\" (UID: \"ae00a62c-0c5b-4e28-ac4b-bfc19709dd99\") " pod="openshift-ingress/payload-processing-e2e-explicit-5c1d26-77c58dd5df-lfjmn" Sep 10 15:57:42.040387 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:42.040364 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae00a62c-0c5b-4e28-ac4b-bfc19709dd99-plugins-config-volume\") pod \"payload-processing-e2e-explicit-5c1d26-77c58dd5df-lfjmn\" (UID: \"ae00a62c-0c5b-4e28-ac4b-bfc19709dd99\") " pod="openshift-ingress/payload-processing-e2e-explicit-5c1d26-77c58dd5df-lfjmn" Sep 10 15:57:42.048878 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:42.048854 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbrtr\" (UniqueName: \"kubernetes.io/projected/ae00a62c-0c5b-4e28-ac4b-bfc19709dd99-kube-api-access-lbrtr\") pod \"payload-processing-e2e-explicit-5c1d26-77c58dd5df-lfjmn\" (UID: \"ae00a62c-0c5b-4e28-ac4b-bfc19709dd99\") " pod="openshift-ingress/payload-processing-e2e-explicit-5c1d26-77c58dd5df-lfjmn" Sep 10 15:57:42.135439 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:42.135354 2576 patch_prober.go:28] interesting pod/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Sep 10 15:57:42.135439 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:42.135403 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2" podUID="2af68ba0-44eb-40fc-b104-ead9c83dcd17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Sep 10 15:57:42.166498 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:42.166471 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-5c1d26-77c58dd5df-lfjmn" Sep 10 15:57:42.467780 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:42.467684 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-5c1d26-77c58dd5df-lfjmn"] Sep 10 15:57:42.470812 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:57:42.470784 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae00a62c_0c5b_4e28_ac4b_bfc19709dd99.slice/crio-b79aabd71fd6f93dd9fdd0d0db4cf3ff23741a61cd91ce17596ce7063dc2c792 WatchSource:0}: Error finding container b79aabd71fd6f93dd9fdd0d0db4cf3ff23741a61cd91ce17596ce7063dc2c792: Status 404 returned error can't find the container with id b79aabd71fd6f93dd9fdd0d0db4cf3ff23741a61cd91ce17596ce7063dc2c792 Sep 10 15:57:42.796443 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:42.796348 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-5c1d26-77c58dd5df-lfjmn" event={"ID":"ae00a62c-0c5b-4e28-ac4b-bfc19709dd99","Type":"ContainerStarted","Data":"0abf8b593899d278c7821dd05059e2f0e519e763016323dcafef494df9dc5cbb"} Sep 10 15:57:42.796443 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:42.796406 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-5c1d26-77c58dd5df-lfjmn" event={"ID":"ae00a62c-0c5b-4e28-ac4b-bfc19709dd99","Type":"ContainerStarted","Data":"b79aabd71fd6f93dd9fdd0d0db4cf3ff23741a61cd91ce17596ce7063dc2c792"} Sep 10 15:57:42.796711 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:42.796478 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-5c1d26-77c58dd5df-lfjmn" Sep 10 15:57:43.135109 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:43.135026 2576 patch_prober.go:28] interesting pod/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Sep 10 15:57:43.135109 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:43.135086 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2" podUID="2af68ba0-44eb-40fc-b104-ead9c83dcd17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Sep 10 15:57:44.134909 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:44.134870 2576 patch_prober.go:28] interesting pod/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Sep 10 15:57:44.135084 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:44.134923 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2" podUID="2af68ba0-44eb-40fc-b104-ead9c83dcd17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Sep 10 15:57:44.433152 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:44.433062 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:45.288512 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:45.288476 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2" Sep 10 15:57:45.289002 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:45.288984 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2" Sep 10 15:57:45.299474 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:45.299452 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2" Sep 10 15:57:45.309509 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:45.309465 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-5c1d26-77c58dd5df-lfjmn" podStartSLOduration=4.309451926 podStartE2EDuration="4.309451926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:57:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:57:42.824761644 +0000 UTC m=+940.178280803" watchObservedRunningTime="2026-09-10 15:57:45.309451926 +0000 UTC m=+942.662971087" Sep 10 15:57:45.480875 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:45.480839 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 15:57:53.733053 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:53.733012 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:53.778021 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:53.777982 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-7a3187a1-6cdc7b6bbf-vv998"] Sep 10 15:57:53.778240 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:53.778208 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-7a3187a1-6cdc7b6bbf-vv998" podUID="d6afd816-e72c-414e-a395-37f788ef3a43" containerName="payload-processing" containerID="cri-o://d70b7c95116ee8aa05a269b8cf6b77afdc739b99cc072bd1d734f7253df42e88" gracePeriod=30 Sep 10 15:57:53.808414 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:53.808384 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-5c1d26-77c58dd5df-lfjmn" Sep 10 15:57:53.830133 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:53.830105 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:53.898802 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:53.898666 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:53.985438 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:53.985351 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:54.039182 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:54.039156 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-7a3187a1-6cdc7b6bbf-vv998" Sep 10 15:57:54.091649 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:54.091617 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:54.158803 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:54.158769 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6afd816-e72c-414e-a395-37f788ef3a43-plugins-config-volume\") pod \"d6afd816-e72c-414e-a395-37f788ef3a43\" (UID: \"d6afd816-e72c-414e-a395-37f788ef3a43\") " Sep 10 15:57:54.159003 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:54.158836 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dnvpc\" (UniqueName: \"kubernetes.io/projected/d6afd816-e72c-414e-a395-37f788ef3a43-kube-api-access-dnvpc\") pod \"d6afd816-e72c-414e-a395-37f788ef3a43\" (UID: \"d6afd816-e72c-414e-a395-37f788ef3a43\") " Sep 10 15:57:54.160094 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:54.160058 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6afd816-e72c-414e-a395-37f788ef3a43-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d6afd816-e72c-414e-a395-37f788ef3a43" (UID: "d6afd816-e72c-414e-a395-37f788ef3a43"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:57:54.162839 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:54.162803 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2"] Sep 10 15:57:54.163100 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:54.163068 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2" podUID="2af68ba0-44eb-40fc-b104-ead9c83dcd17" containerName="istio-proxy" containerID="cri-o://c415822991484e3bfcd696e6878f6184aa0fccca3d551b567181210b80c1ac61" gracePeriod=30 Sep 10 15:57:54.163779 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:54.163724 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6afd816-e72c-414e-a395-37f788ef3a43-kube-api-access-dnvpc" (OuterVolumeSpecName: "kube-api-access-dnvpc") pod "d6afd816-e72c-414e-a395-37f788ef3a43" (UID: "d6afd816-e72c-414e-a395-37f788ef3a43"). InnerVolumeSpecName "kube-api-access-dnvpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:57:54.239601 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:54.239506 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:54.261883 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:54.261386 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6afd816-e72c-414e-a395-37f788ef3a43-plugins-config-volume\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:57:54.261883 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:54.261422 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnvpc\" (UniqueName: \"kubernetes.io/projected/d6afd816-e72c-414e-a395-37f788ef3a43-kube-api-access-dnvpc\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:57:54.809292 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:54.809250 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:54.868852 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:54.868815 2576 generic.go:356] "Generic (PLEG): container finished" podID="d6afd816-e72c-414e-a395-37f788ef3a43" containerID="d70b7c95116ee8aa05a269b8cf6b77afdc739b99cc072bd1d734f7253df42e88" exitCode=0 Sep 10 15:57:54.869040 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:54.868857 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-7a3187a1-6cdc7b6bbf-vv998" Sep 10 15:57:54.869040 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:54.868913 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-7a3187a1-6cdc7b6bbf-vv998" event={"ID":"d6afd816-e72c-414e-a395-37f788ef3a43","Type":"ContainerDied","Data":"d70b7c95116ee8aa05a269b8cf6b77afdc739b99cc072bd1d734f7253df42e88"} Sep 10 15:57:54.869040 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:54.868968 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-7a3187a1-6cdc7b6bbf-vv998" event={"ID":"d6afd816-e72c-414e-a395-37f788ef3a43","Type":"ContainerDied","Data":"1e490a50fdc0c004aaf9f7d6c7283adffc1448035518a4401643d974652f03e4"} Sep 10 15:57:54.869040 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:54.868995 2576 scope.go:117] "RemoveContainer" containerID="d70b7c95116ee8aa05a269b8cf6b77afdc739b99cc072bd1d734f7253df42e88" Sep 10 15:57:54.878975 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:54.878953 2576 scope.go:117] "RemoveContainer" containerID="d70b7c95116ee8aa05a269b8cf6b77afdc739b99cc072bd1d734f7253df42e88" Sep 10 15:57:54.879281 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:57:54.879263 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d70b7c95116ee8aa05a269b8cf6b77afdc739b99cc072bd1d734f7253df42e88\": container with ID starting with d70b7c95116ee8aa05a269b8cf6b77afdc739b99cc072bd1d734f7253df42e88 not found: ID does not exist" containerID="d70b7c95116ee8aa05a269b8cf6b77afdc739b99cc072bd1d734f7253df42e88" Sep 10 15:57:54.879327 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:54.879291 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d70b7c95116ee8aa05a269b8cf6b77afdc739b99cc072bd1d734f7253df42e88"} err="failed to get container status \"d70b7c95116ee8aa05a269b8cf6b77afdc739b99cc072bd1d734f7253df42e88\": rpc error: code = NotFound desc = could not find container \"d70b7c95116ee8aa05a269b8cf6b77afdc739b99cc072bd1d734f7253df42e88\": container with ID starting with d70b7c95116ee8aa05a269b8cf6b77afdc739b99cc072bd1d734f7253df42e88 not found: ID does not exist" Sep 10 15:57:54.890375 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:54.890346 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-7a3187a1-6cdc7b6bbf-vv998"] Sep 10 15:57:54.892618 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:54.892590 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:54.892882 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:54.892864 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-7a3187a1-6cdc7b6bbf-vv998"] Sep 10 15:57:54.960799 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:54.960756 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:55.169380 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:55.169295 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6afd816-e72c-414e-a395-37f788ef3a43" path="/var/lib/kubelet/pods/d6afd816-e72c-414e-a395-37f788ef3a43/volumes" Sep 10 15:57:56.345841 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:56.345810 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:59.452587 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:59.452557 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2" Sep 10 15:57:59.510662 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:59.510635 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2af68ba0-44eb-40fc-b104-ead9c83dcd17-istiod-ca-cert\") pod \"2af68ba0-44eb-40fc-b104-ead9c83dcd17\" (UID: \"2af68ba0-44eb-40fc-b104-ead9c83dcd17\") " Sep 10 15:57:59.510843 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:59.510692 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2af68ba0-44eb-40fc-b104-ead9c83dcd17-istio-token\") pod \"2af68ba0-44eb-40fc-b104-ead9c83dcd17\" (UID: \"2af68ba0-44eb-40fc-b104-ead9c83dcd17\") " Sep 10 15:57:59.510843 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:59.510720 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2af68ba0-44eb-40fc-b104-ead9c83dcd17-credential-socket\") pod \"2af68ba0-44eb-40fc-b104-ead9c83dcd17\" (UID: \"2af68ba0-44eb-40fc-b104-ead9c83dcd17\") " Sep 10 15:57:59.510843 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:59.510802 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2af68ba0-44eb-40fc-b104-ead9c83dcd17-istio-data\") pod \"2af68ba0-44eb-40fc-b104-ead9c83dcd17\" (UID: \"2af68ba0-44eb-40fc-b104-ead9c83dcd17\") " Sep 10 15:57:59.511018 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:59.510876 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2af68ba0-44eb-40fc-b104-ead9c83dcd17-workload-socket\") pod \"2af68ba0-44eb-40fc-b104-ead9c83dcd17\" (UID: \"2af68ba0-44eb-40fc-b104-ead9c83dcd17\") " Sep 10 15:57:59.511018 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:59.510910 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2af68ba0-44eb-40fc-b104-ead9c83dcd17-istio-envoy\") pod \"2af68ba0-44eb-40fc-b104-ead9c83dcd17\" (UID: \"2af68ba0-44eb-40fc-b104-ead9c83dcd17\") " Sep 10 15:57:59.511018 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:59.510958 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rs6gb\" (UniqueName: \"kubernetes.io/projected/2af68ba0-44eb-40fc-b104-ead9c83dcd17-kube-api-access-rs6gb\") pod \"2af68ba0-44eb-40fc-b104-ead9c83dcd17\" (UID: \"2af68ba0-44eb-40fc-b104-ead9c83dcd17\") " Sep 10 15:57:59.511018 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:59.510999 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2af68ba0-44eb-40fc-b104-ead9c83dcd17-workload-certs\") pod \"2af68ba0-44eb-40fc-b104-ead9c83dcd17\" (UID: \"2af68ba0-44eb-40fc-b104-ead9c83dcd17\") " Sep 10 15:57:59.511206 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:59.511030 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2af68ba0-44eb-40fc-b104-ead9c83dcd17-istio-podinfo\") pod \"2af68ba0-44eb-40fc-b104-ead9c83dcd17\" (UID: \"2af68ba0-44eb-40fc-b104-ead9c83dcd17\") " Sep 10 15:57:59.511206 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:59.511036 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2af68ba0-44eb-40fc-b104-ead9c83dcd17-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2af68ba0-44eb-40fc-b104-ead9c83dcd17" (UID: "2af68ba0-44eb-40fc-b104-ead9c83dcd17"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:57:59.511206 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:59.511047 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2af68ba0-44eb-40fc-b104-ead9c83dcd17-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2af68ba0-44eb-40fc-b104-ead9c83dcd17" (UID: "2af68ba0-44eb-40fc-b104-ead9c83dcd17"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:57:59.511363 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:59.511202 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2af68ba0-44eb-40fc-b104-ead9c83dcd17-istio-data" (OuterVolumeSpecName: "istio-data") pod "2af68ba0-44eb-40fc-b104-ead9c83dcd17" (UID: "2af68ba0-44eb-40fc-b104-ead9c83dcd17"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:57:59.511429 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:59.511414 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2af68ba0-44eb-40fc-b104-ead9c83dcd17-istiod-ca-cert\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:57:59.511495 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:59.511433 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2af68ba0-44eb-40fc-b104-ead9c83dcd17-credential-socket\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:57:59.511495 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:59.511447 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2af68ba0-44eb-40fc-b104-ead9c83dcd17-istio-data\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:57:59.511659 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:59.511632 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2af68ba0-44eb-40fc-b104-ead9c83dcd17-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2af68ba0-44eb-40fc-b104-ead9c83dcd17" (UID: "2af68ba0-44eb-40fc-b104-ead9c83dcd17"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:57:59.511928 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:59.511861 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2af68ba0-44eb-40fc-b104-ead9c83dcd17-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2af68ba0-44eb-40fc-b104-ead9c83dcd17" (UID: "2af68ba0-44eb-40fc-b104-ead9c83dcd17"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:57:59.513623 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:59.513589 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2af68ba0-44eb-40fc-b104-ead9c83dcd17-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2af68ba0-44eb-40fc-b104-ead9c83dcd17" (UID: "2af68ba0-44eb-40fc-b104-ead9c83dcd17"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 15:57:59.514162 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:59.514137 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2af68ba0-44eb-40fc-b104-ead9c83dcd17-kube-api-access-rs6gb" (OuterVolumeSpecName: "kube-api-access-rs6gb") pod "2af68ba0-44eb-40fc-b104-ead9c83dcd17" (UID: "2af68ba0-44eb-40fc-b104-ead9c83dcd17"). InnerVolumeSpecName "kube-api-access-rs6gb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:57:59.514260 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:59.514166 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2af68ba0-44eb-40fc-b104-ead9c83dcd17-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2af68ba0-44eb-40fc-b104-ead9c83dcd17" (UID: "2af68ba0-44eb-40fc-b104-ead9c83dcd17"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:57:59.514446 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:59.514412 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2af68ba0-44eb-40fc-b104-ead9c83dcd17-istio-token" (OuterVolumeSpecName: "istio-token") pod "2af68ba0-44eb-40fc-b104-ead9c83dcd17" (UID: "2af68ba0-44eb-40fc-b104-ead9c83dcd17"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:57:59.612063 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:59.612028 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2af68ba0-44eb-40fc-b104-ead9c83dcd17-workload-socket\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:57:59.612063 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:59.612062 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2af68ba0-44eb-40fc-b104-ead9c83dcd17-istio-envoy\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:57:59.612222 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:59.612078 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs6gb\" (UniqueName: \"kubernetes.io/projected/2af68ba0-44eb-40fc-b104-ead9c83dcd17-kube-api-access-rs6gb\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:57:59.612222 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:59.612091 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2af68ba0-44eb-40fc-b104-ead9c83dcd17-workload-certs\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:57:59.612222 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:59.612104 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2af68ba0-44eb-40fc-b104-ead9c83dcd17-istio-podinfo\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:57:59.612222 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:59.612116 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2af68ba0-44eb-40fc-b104-ead9c83dcd17-istio-token\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:57:59.897622 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:59.897586 2576 generic.go:356] "Generic (PLEG): container finished" podID="2af68ba0-44eb-40fc-b104-ead9c83dcd17" containerID="c415822991484e3bfcd696e6878f6184aa0fccca3d551b567181210b80c1ac61" exitCode=0 Sep 10 15:57:59.897815 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:59.897629 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2" event={"ID":"2af68ba0-44eb-40fc-b104-ead9c83dcd17","Type":"ContainerDied","Data":"c415822991484e3bfcd696e6878f6184aa0fccca3d551b567181210b80c1ac61"} Sep 10 15:57:59.897815 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:59.897648 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2" Sep 10 15:57:59.897815 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:59.897688 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2" event={"ID":"2af68ba0-44eb-40fc-b104-ead9c83dcd17","Type":"ContainerDied","Data":"e3d85ca975b64ba6651fd889985c80f267f6101f6606c60d4d085d0b92aa9018"} Sep 10 15:57:59.897815 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:59.897711 2576 scope.go:117] "RemoveContainer" containerID="c415822991484e3bfcd696e6878f6184aa0fccca3d551b567181210b80c1ac61" Sep 10 15:57:59.907308 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:59.907290 2576 scope.go:117] "RemoveContainer" containerID="9a29c5a8623d1137d00fcb1de40e828c7d0ec6045a3bae5a5fb774cd7b4d63ef" Sep 10 15:57:59.915187 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:59.915170 2576 scope.go:117] "RemoveContainer" containerID="c415822991484e3bfcd696e6878f6184aa0fccca3d551b567181210b80c1ac61" Sep 10 15:57:59.915440 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:57:59.915423 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c415822991484e3bfcd696e6878f6184aa0fccca3d551b567181210b80c1ac61\": container with ID starting with c415822991484e3bfcd696e6878f6184aa0fccca3d551b567181210b80c1ac61 not found: ID does not exist" containerID="c415822991484e3bfcd696e6878f6184aa0fccca3d551b567181210b80c1ac61" Sep 10 15:57:59.915510 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:59.915452 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c415822991484e3bfcd696e6878f6184aa0fccca3d551b567181210b80c1ac61"} err="failed to get container status \"c415822991484e3bfcd696e6878f6184aa0fccca3d551b567181210b80c1ac61\": rpc error: code = NotFound desc = could not find container \"c415822991484e3bfcd696e6878f6184aa0fccca3d551b567181210b80c1ac61\": container with ID starting with c415822991484e3bfcd696e6878f6184aa0fccca3d551b567181210b80c1ac61 not found: ID does not exist" Sep 10 15:57:59.915510 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:59.915476 2576 scope.go:117] "RemoveContainer" containerID="9a29c5a8623d1137d00fcb1de40e828c7d0ec6045a3bae5a5fb774cd7b4d63ef" Sep 10 15:57:59.915721 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:57:59.915706 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a29c5a8623d1137d00fcb1de40e828c7d0ec6045a3bae5a5fb774cd7b4d63ef\": container with ID starting with 9a29c5a8623d1137d00fcb1de40e828c7d0ec6045a3bae5a5fb774cd7b4d63ef not found: ID does not exist" containerID="9a29c5a8623d1137d00fcb1de40e828c7d0ec6045a3bae5a5fb774cd7b4d63ef" Sep 10 15:57:59.915784 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:59.915726 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a29c5a8623d1137d00fcb1de40e828c7d0ec6045a3bae5a5fb774cd7b4d63ef"} err="failed to get container status \"9a29c5a8623d1137d00fcb1de40e828c7d0ec6045a3bae5a5fb774cd7b4d63ef\": rpc error: code = NotFound desc = could not find container \"9a29c5a8623d1137d00fcb1de40e828c7d0ec6045a3bae5a5fb774cd7b4d63ef\": container with ID starting with 9a29c5a8623d1137d00fcb1de40e828c7d0ec6045a3bae5a5fb774cd7b4d63ef not found: ID does not exist" Sep 10 15:57:59.921172 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:59.921153 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2"] Sep 10 15:57:59.926157 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:57:59.926136 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-7a3187a1-openshift-default-567f576dbd-5jhk2"] Sep 10 15:58:00.779138 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:00.779105 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:01.169896 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:01.169865 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2af68ba0-44eb-40fc-b104-ead9c83dcd17" path="/var/lib/kubelet/pods/2af68ba0-44eb-40fc-b104-ead9c83dcd17/volumes" Sep 10 15:58:02.267164 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:02.267124 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:02.369404 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:02.369352 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:02.484181 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:02.484145 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:02.604293 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:02.604263 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:03.003085 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:03.003001 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:03.356609 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:03.356576 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:04.636337 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:04.636290 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:07.590446 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:07.590413 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 15:58:23.487569 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:23.487536 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:24.069975 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:24.069936 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 15:58:25.313087 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:25.313050 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:25.372366 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:25.372317 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:28.907487 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:28.907449 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:35.219563 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:35.219528 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:35.366522 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:35.366488 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:35.558169 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:35.558139 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:36.083939 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:36.083912 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77585476db-g2dkb"] Sep 10 15:58:36.233979 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:36.233946 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:36.333488 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:36.333460 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:36.434347 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:36.434280 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:36.562098 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:36.562037 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:36.623809 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:36.623779 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5c1d26-bvx66"] Sep 10 15:58:36.624227 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:36.624213 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6afd816-e72c-414e-a395-37f788ef3a43" containerName="payload-processing" Sep 10 15:58:36.624272 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:36.624229 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6afd816-e72c-414e-a395-37f788ef3a43" containerName="payload-processing" Sep 10 15:58:36.624272 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:36.624238 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2af68ba0-44eb-40fc-b104-ead9c83dcd17" containerName="istio-proxy" Sep 10 15:58:36.624272 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:36.624245 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2af68ba0-44eb-40fc-b104-ead9c83dcd17" containerName="istio-proxy" Sep 10 15:58:36.624272 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:36.624255 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="780ebc58-fc9c-46d3-b893-6bff1e70a445" containerName="istio-proxy" Sep 10 15:58:36.624272 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:36.624260 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="780ebc58-fc9c-46d3-b893-6bff1e70a445" containerName="istio-proxy" Sep 10 15:58:36.624433 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:36.624331 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6afd816-e72c-414e-a395-37f788ef3a43" containerName="payload-processing" Sep 10 15:58:36.624433 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:36.624337 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2af68ba0-44eb-40fc-b104-ead9c83dcd17" containerName="istio-proxy" Sep 10 15:58:36.624433 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:36.624343 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2af68ba0-44eb-40fc-b104-ead9c83dcd17" containerName="istio-proxy" Sep 10 15:58:36.627031 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:36.627014 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5c1d26-bvx66" Sep 10 15:58:36.629435 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:36.629414 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 15:58:36.629514 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:36.629417 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwwht\"" Sep 10 15:58:36.636985 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:36.636961 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5c1d26-bvx66"] Sep 10 15:58:36.753501 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:36.753429 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1829c69d-deb6-4ba1-8493-f981d59951b7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-5c1d26-bvx66\" (UID: \"1829c69d-deb6-4ba1-8493-f981d59951b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5c1d26-bvx66" Sep 10 15:58:36.854814 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:36.854787 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1829c69d-deb6-4ba1-8493-f981d59951b7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-5c1d26-bvx66\" (UID: \"1829c69d-deb6-4ba1-8493-f981d59951b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5c1d26-bvx66" Sep 10 15:58:36.855364 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:36.855345 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1829c69d-deb6-4ba1-8493-f981d59951b7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-5c1d26-bvx66\" (UID: \"1829c69d-deb6-4ba1-8493-f981d59951b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5c1d26-bvx66" Sep 10 15:58:36.939393 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:36.939367 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5c1d26-bvx66" Sep 10 15:58:37.212312 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:37.212284 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5c1d26-bvx66"] Sep 10 15:58:37.213767 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:58:37.213718 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1829c69d_deb6_4ba1_8493_f981d59951b7.slice/crio-1b73a860fb2c19ee71083cd1e1d82edd422094bd1d1673bd89c1b365071bb0a1 WatchSource:0}: Error finding container 1b73a860fb2c19ee71083cd1e1d82edd422094bd1d1673bd89c1b365071bb0a1: Status 404 returned error can't find the container with id 1b73a860fb2c19ee71083cd1e1d82edd422094bd1d1673bd89c1b365071bb0a1 Sep 10 15:58:38.096323 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:38.096232 2576 generic.go:356] "Generic (PLEG): container finished" podID="1829c69d-deb6-4ba1-8493-f981d59951b7" containerID="238a3f729d7dbd156883f2f7fa8218eceb437d40fec279be77c3bbaed3055c83" exitCode=0 Sep 10 15:58:38.096323 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:38.096291 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5c1d26-bvx66" event={"ID":"1829c69d-deb6-4ba1-8493-f981d59951b7","Type":"ContainerDied","Data":"238a3f729d7dbd156883f2f7fa8218eceb437d40fec279be77c3bbaed3055c83"} Sep 10 15:58:38.096714 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:38.096340 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5c1d26-bvx66" event={"ID":"1829c69d-deb6-4ba1-8493-f981d59951b7","Type":"ContainerStarted","Data":"1b73a860fb2c19ee71083cd1e1d82edd422094bd1d1673bd89c1b365071bb0a1"} Sep 10 15:58:39.234158 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:39.234134 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5c1d26-bvx66" Sep 10 15:58:39.275887 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:39.275861 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1829c69d-deb6-4ba1-8493-f981d59951b7-maas-api-service-ca\") pod \"1829c69d-deb6-4ba1-8493-f981d59951b7\" (UID: \"1829c69d-deb6-4ba1-8493-f981d59951b7\") " Sep 10 15:58:39.276279 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:39.276253 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1829c69d-deb6-4ba1-8493-f981d59951b7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1829c69d-deb6-4ba1-8493-f981d59951b7" (UID: "1829c69d-deb6-4ba1-8493-f981d59951b7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:58:39.377007 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:39.376923 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1829c69d-deb6-4ba1-8493-f981d59951b7-maas-api-service-ca\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:58:40.107950 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:40.107916 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5c1d26-bvx66" event={"ID":"1829c69d-deb6-4ba1-8493-f981d59951b7","Type":"ContainerDied","Data":"1b73a860fb2c19ee71083cd1e1d82edd422094bd1d1673bd89c1b365071bb0a1"} Sep 10 15:58:40.108126 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:40.107964 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b73a860fb2c19ee71083cd1e1d82edd422094bd1d1673bd89c1b365071bb0a1" Sep 10 15:58:40.108126 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:40.107927 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5c1d26-bvx66" Sep 10 15:58:41.483242 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:41.483206 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 15:58:47.268812 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:47.268772 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:47.302441 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:47.302412 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-5c1d26-77c58dd5df-lfjmn"] Sep 10 15:58:47.302953 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:47.302919 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-5c1d26-77c58dd5df-lfjmn" podUID="ae00a62c-0c5b-4e28-ac4b-bfc19709dd99" containerName="payload-processing" containerID="cri-o://0abf8b593899d278c7821dd05059e2f0e519e763016323dcafef494df9dc5cbb" gracePeriod=30 Sep 10 15:58:47.371551 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:47.371518 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:47.475416 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:47.475378 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:47.559683 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:47.559651 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5c1d26-bvx66"] Sep 10 15:58:47.564135 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:47.564111 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5c1d26-bvx66"] Sep 10 15:58:47.580801 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:47.580770 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:47.584560 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:47.584539 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-5c1d26-77c58dd5df-lfjmn" Sep 10 15:58:47.649878 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:47.649853 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae00a62c-0c5b-4e28-ac4b-bfc19709dd99-plugins-config-volume\") pod \"ae00a62c-0c5b-4e28-ac4b-bfc19709dd99\" (UID: \"ae00a62c-0c5b-4e28-ac4b-bfc19709dd99\") " Sep 10 15:58:47.650080 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:47.649947 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lbrtr\" (UniqueName: \"kubernetes.io/projected/ae00a62c-0c5b-4e28-ac4b-bfc19709dd99-kube-api-access-lbrtr\") pod \"ae00a62c-0c5b-4e28-ac4b-bfc19709dd99\" (UID: \"ae00a62c-0c5b-4e28-ac4b-bfc19709dd99\") " Sep 10 15:58:47.650243 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:47.650219 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae00a62c-0c5b-4e28-ac4b-bfc19709dd99-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ae00a62c-0c5b-4e28-ac4b-bfc19709dd99" (UID: "ae00a62c-0c5b-4e28-ac4b-bfc19709dd99"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:58:47.650343 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:47.650327 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae00a62c-0c5b-4e28-ac4b-bfc19709dd99-plugins-config-volume\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:58:47.652270 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:47.652240 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae00a62c-0c5b-4e28-ac4b-bfc19709dd99-kube-api-access-lbrtr" (OuterVolumeSpecName: "kube-api-access-lbrtr") pod "ae00a62c-0c5b-4e28-ac4b-bfc19709dd99" (UID: "ae00a62c-0c5b-4e28-ac4b-bfc19709dd99"). InnerVolumeSpecName "kube-api-access-lbrtr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:58:47.751140 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:47.751105 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbrtr\" (UniqueName: \"kubernetes.io/projected/ae00a62c-0c5b-4e28-ac4b-bfc19709dd99-kube-api-access-lbrtr\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:58:48.029905 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:48.029808 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:48.151537 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:48.151507 2576 generic.go:356] "Generic (PLEG): container finished" podID="ae00a62c-0c5b-4e28-ac4b-bfc19709dd99" containerID="0abf8b593899d278c7821dd05059e2f0e519e763016323dcafef494df9dc5cbb" exitCode=0 Sep 10 15:58:48.151699 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:48.151553 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-5c1d26-77c58dd5df-lfjmn" Sep 10 15:58:48.151699 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:48.151560 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-5c1d26-77c58dd5df-lfjmn" event={"ID":"ae00a62c-0c5b-4e28-ac4b-bfc19709dd99","Type":"ContainerDied","Data":"0abf8b593899d278c7821dd05059e2f0e519e763016323dcafef494df9dc5cbb"} Sep 10 15:58:48.151699 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:48.151607 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-5c1d26-77c58dd5df-lfjmn" event={"ID":"ae00a62c-0c5b-4e28-ac4b-bfc19709dd99","Type":"ContainerDied","Data":"b79aabd71fd6f93dd9fdd0d0db4cf3ff23741a61cd91ce17596ce7063dc2c792"} Sep 10 15:58:48.151699 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:48.151635 2576 scope.go:117] "RemoveContainer" containerID="0abf8b593899d278c7821dd05059e2f0e519e763016323dcafef494df9dc5cbb" Sep 10 15:58:48.166288 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:48.166262 2576 scope.go:117] "RemoveContainer" containerID="0abf8b593899d278c7821dd05059e2f0e519e763016323dcafef494df9dc5cbb" Sep 10 15:58:48.166588 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:58:48.166561 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0abf8b593899d278c7821dd05059e2f0e519e763016323dcafef494df9dc5cbb\": container with ID starting with 0abf8b593899d278c7821dd05059e2f0e519e763016323dcafef494df9dc5cbb not found: ID does not exist" containerID="0abf8b593899d278c7821dd05059e2f0e519e763016323dcafef494df9dc5cbb" Sep 10 15:58:48.166641 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:48.166596 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0abf8b593899d278c7821dd05059e2f0e519e763016323dcafef494df9dc5cbb"} err="failed to get container status \"0abf8b593899d278c7821dd05059e2f0e519e763016323dcafef494df9dc5cbb\": rpc error: code = NotFound desc = could not find container \"0abf8b593899d278c7821dd05059e2f0e519e763016323dcafef494df9dc5cbb\": container with ID starting with 0abf8b593899d278c7821dd05059e2f0e519e763016323dcafef494df9dc5cbb not found: ID does not exist" Sep 10 15:58:48.185193 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:48.185169 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-5c1d26-77c58dd5df-lfjmn"] Sep 10 15:58:48.192969 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:48.192946 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-5c1d26-77c58dd5df-lfjmn"] Sep 10 15:58:49.168509 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:49.168475 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1829c69d-deb6-4ba1-8493-f981d59951b7" path="/var/lib/kubelet/pods/1829c69d-deb6-4ba1-8493-f981d59951b7/volumes" Sep 10 15:58:49.168889 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:49.168876 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae00a62c-0c5b-4e28-ac4b-bfc19709dd99" path="/var/lib/kubelet/pods/ae00a62c-0c5b-4e28-ac4b-bfc19709dd99/volumes" Sep 10 15:58:51.402920 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:51.402888 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:52.202272 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:52.202240 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 15:58:54.825811 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:54.825773 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:54.935365 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:54.935328 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:55.147472 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:55.147376 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:55.593311 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:55.593275 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:55.909976 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:55.909891 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-066ed8-openshift-default-547b48b74-ztbdw"] Sep 10 15:58:55.910481 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:55.910460 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1829c69d-deb6-4ba1-8493-f981d59951b7" containerName="revoke-keys" Sep 10 15:58:55.910565 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:55.910484 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1829c69d-deb6-4ba1-8493-f981d59951b7" containerName="revoke-keys" Sep 10 15:58:55.910565 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:55.910528 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2af68ba0-44eb-40fc-b104-ead9c83dcd17" containerName="istio-proxy" Sep 10 15:58:55.910565 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:55.910537 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2af68ba0-44eb-40fc-b104-ead9c83dcd17" containerName="istio-proxy" Sep 10 15:58:55.910565 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:55.910548 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae00a62c-0c5b-4e28-ac4b-bfc19709dd99" containerName="payload-processing" Sep 10 15:58:55.910565 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:55.910557 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae00a62c-0c5b-4e28-ac4b-bfc19709dd99" containerName="payload-processing" Sep 10 15:58:55.910847 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:55.910663 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1829c69d-deb6-4ba1-8493-f981d59951b7" containerName="revoke-keys" Sep 10 15:58:55.910847 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:55.910680 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae00a62c-0c5b-4e28-ac4b-bfc19709dd99" containerName="payload-processing" Sep 10 15:58:55.915243 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:55.915221 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-066ed8-openshift-default-547b48b74-ztbdw" Sep 10 15:58:55.918808 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:55.918788 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-066ed8-openshift-default-dockercfg-lkf7q\"" Sep 10 15:58:55.932914 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:55.932887 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-066ed8-openshift-default-547b48b74-ztbdw"] Sep 10 15:58:55.953965 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:55.953943 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:56.023445 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:56.023399 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4524090d-1ce2-4bef-b917-483a89219247-credential-socket\") pod \"e2e-noten-066ed8-openshift-default-547b48b74-ztbdw\" (UID: \"4524090d-1ce2-4bef-b917-483a89219247\") " pod="openshift-ingress/e2e-noten-066ed8-openshift-default-547b48b74-ztbdw" Sep 10 15:58:56.023641 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:56.023492 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4524090d-1ce2-4bef-b917-483a89219247-workload-socket\") pod \"e2e-noten-066ed8-openshift-default-547b48b74-ztbdw\" (UID: \"4524090d-1ce2-4bef-b917-483a89219247\") " pod="openshift-ingress/e2e-noten-066ed8-openshift-default-547b48b74-ztbdw" Sep 10 15:58:56.023641 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:56.023570 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4524090d-1ce2-4bef-b917-483a89219247-istiod-ca-cert\") pod \"e2e-noten-066ed8-openshift-default-547b48b74-ztbdw\" (UID: \"4524090d-1ce2-4bef-b917-483a89219247\") " pod="openshift-ingress/e2e-noten-066ed8-openshift-default-547b48b74-ztbdw" Sep 10 15:58:56.023641 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:56.023599 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5lmr\" (UniqueName: \"kubernetes.io/projected/4524090d-1ce2-4bef-b917-483a89219247-kube-api-access-p5lmr\") pod \"e2e-noten-066ed8-openshift-default-547b48b74-ztbdw\" (UID: \"4524090d-1ce2-4bef-b917-483a89219247\") " pod="openshift-ingress/e2e-noten-066ed8-openshift-default-547b48b74-ztbdw" Sep 10 15:58:56.023641 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:56.023625 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4524090d-1ce2-4bef-b917-483a89219247-istio-podinfo\") pod \"e2e-noten-066ed8-openshift-default-547b48b74-ztbdw\" (UID: \"4524090d-1ce2-4bef-b917-483a89219247\") " pod="openshift-ingress/e2e-noten-066ed8-openshift-default-547b48b74-ztbdw" Sep 10 15:58:56.023927 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:56.023651 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4524090d-1ce2-4bef-b917-483a89219247-istio-token\") pod \"e2e-noten-066ed8-openshift-default-547b48b74-ztbdw\" (UID: \"4524090d-1ce2-4bef-b917-483a89219247\") " pod="openshift-ingress/e2e-noten-066ed8-openshift-default-547b48b74-ztbdw" Sep 10 15:58:56.023927 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:56.023704 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4524090d-1ce2-4bef-b917-483a89219247-istio-envoy\") pod \"e2e-noten-066ed8-openshift-default-547b48b74-ztbdw\" (UID: \"4524090d-1ce2-4bef-b917-483a89219247\") " pod="openshift-ingress/e2e-noten-066ed8-openshift-default-547b48b74-ztbdw" Sep 10 15:58:56.023927 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:56.023756 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4524090d-1ce2-4bef-b917-483a89219247-istio-data\") pod \"e2e-noten-066ed8-openshift-default-547b48b74-ztbdw\" (UID: \"4524090d-1ce2-4bef-b917-483a89219247\") " pod="openshift-ingress/e2e-noten-066ed8-openshift-default-547b48b74-ztbdw" Sep 10 15:58:56.023927 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:56.023808 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4524090d-1ce2-4bef-b917-483a89219247-workload-certs\") pod \"e2e-noten-066ed8-openshift-default-547b48b74-ztbdw\" (UID: \"4524090d-1ce2-4bef-b917-483a89219247\") " pod="openshift-ingress/e2e-noten-066ed8-openshift-default-547b48b74-ztbdw" Sep 10 15:58:56.125054 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:56.125018 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4524090d-1ce2-4bef-b917-483a89219247-istiod-ca-cert\") pod \"e2e-noten-066ed8-openshift-default-547b48b74-ztbdw\" (UID: \"4524090d-1ce2-4bef-b917-483a89219247\") " pod="openshift-ingress/e2e-noten-066ed8-openshift-default-547b48b74-ztbdw" Sep 10 15:58:56.125054 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:56.125056 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5lmr\" (UniqueName: \"kubernetes.io/projected/4524090d-1ce2-4bef-b917-483a89219247-kube-api-access-p5lmr\") pod \"e2e-noten-066ed8-openshift-default-547b48b74-ztbdw\" (UID: \"4524090d-1ce2-4bef-b917-483a89219247\") " pod="openshift-ingress/e2e-noten-066ed8-openshift-default-547b48b74-ztbdw" Sep 10 15:58:56.125300 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:56.125083 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4524090d-1ce2-4bef-b917-483a89219247-istio-podinfo\") pod \"e2e-noten-066ed8-openshift-default-547b48b74-ztbdw\" (UID: \"4524090d-1ce2-4bef-b917-483a89219247\") " pod="openshift-ingress/e2e-noten-066ed8-openshift-default-547b48b74-ztbdw" Sep 10 15:58:56.125300 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:56.125205 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4524090d-1ce2-4bef-b917-483a89219247-istio-token\") pod \"e2e-noten-066ed8-openshift-default-547b48b74-ztbdw\" (UID: \"4524090d-1ce2-4bef-b917-483a89219247\") " pod="openshift-ingress/e2e-noten-066ed8-openshift-default-547b48b74-ztbdw" Sep 10 15:58:56.125300 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:56.125273 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4524090d-1ce2-4bef-b917-483a89219247-istio-envoy\") pod \"e2e-noten-066ed8-openshift-default-547b48b74-ztbdw\" (UID: \"4524090d-1ce2-4bef-b917-483a89219247\") " pod="openshift-ingress/e2e-noten-066ed8-openshift-default-547b48b74-ztbdw" Sep 10 15:58:56.125512 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:56.125307 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4524090d-1ce2-4bef-b917-483a89219247-istio-data\") pod \"e2e-noten-066ed8-openshift-default-547b48b74-ztbdw\" (UID: \"4524090d-1ce2-4bef-b917-483a89219247\") " pod="openshift-ingress/e2e-noten-066ed8-openshift-default-547b48b74-ztbdw" Sep 10 15:58:56.125512 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:56.125356 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4524090d-1ce2-4bef-b917-483a89219247-workload-certs\") pod \"e2e-noten-066ed8-openshift-default-547b48b74-ztbdw\" (UID: \"4524090d-1ce2-4bef-b917-483a89219247\") " pod="openshift-ingress/e2e-noten-066ed8-openshift-default-547b48b74-ztbdw" Sep 10 15:58:56.125512 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:56.125403 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4524090d-1ce2-4bef-b917-483a89219247-credential-socket\") pod \"e2e-noten-066ed8-openshift-default-547b48b74-ztbdw\" (UID: \"4524090d-1ce2-4bef-b917-483a89219247\") " pod="openshift-ingress/e2e-noten-066ed8-openshift-default-547b48b74-ztbdw" Sep 10 15:58:56.125512 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:56.125473 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4524090d-1ce2-4bef-b917-483a89219247-workload-socket\") pod \"e2e-noten-066ed8-openshift-default-547b48b74-ztbdw\" (UID: \"4524090d-1ce2-4bef-b917-483a89219247\") " pod="openshift-ingress/e2e-noten-066ed8-openshift-default-547b48b74-ztbdw" Sep 10 15:58:56.125818 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:56.125783 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4524090d-1ce2-4bef-b917-483a89219247-workload-certs\") pod \"e2e-noten-066ed8-openshift-default-547b48b74-ztbdw\" (UID: \"4524090d-1ce2-4bef-b917-483a89219247\") " pod="openshift-ingress/e2e-noten-066ed8-openshift-default-547b48b74-ztbdw" Sep 10 15:58:56.125902 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:56.125821 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4524090d-1ce2-4bef-b917-483a89219247-istio-data\") pod \"e2e-noten-066ed8-openshift-default-547b48b74-ztbdw\" (UID: \"4524090d-1ce2-4bef-b917-483a89219247\") " pod="openshift-ingress/e2e-noten-066ed8-openshift-default-547b48b74-ztbdw" Sep 10 15:58:56.125902 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:56.125880 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4524090d-1ce2-4bef-b917-483a89219247-workload-socket\") pod \"e2e-noten-066ed8-openshift-default-547b48b74-ztbdw\" (UID: \"4524090d-1ce2-4bef-b917-483a89219247\") " pod="openshift-ingress/e2e-noten-066ed8-openshift-default-547b48b74-ztbdw" Sep 10 15:58:56.125982 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:56.125908 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4524090d-1ce2-4bef-b917-483a89219247-credential-socket\") pod \"e2e-noten-066ed8-openshift-default-547b48b74-ztbdw\" (UID: \"4524090d-1ce2-4bef-b917-483a89219247\") " pod="openshift-ingress/e2e-noten-066ed8-openshift-default-547b48b74-ztbdw" Sep 10 15:58:56.126065 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:56.126044 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4524090d-1ce2-4bef-b917-483a89219247-istiod-ca-cert\") pod \"e2e-noten-066ed8-openshift-default-547b48b74-ztbdw\" (UID: \"4524090d-1ce2-4bef-b917-483a89219247\") " pod="openshift-ingress/e2e-noten-066ed8-openshift-default-547b48b74-ztbdw" Sep 10 15:58:56.127686 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:56.127668 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4524090d-1ce2-4bef-b917-483a89219247-istio-envoy\") pod \"e2e-noten-066ed8-openshift-default-547b48b74-ztbdw\" (UID: \"4524090d-1ce2-4bef-b917-483a89219247\") " pod="openshift-ingress/e2e-noten-066ed8-openshift-default-547b48b74-ztbdw" Sep 10 15:58:56.127764 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:56.127699 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4524090d-1ce2-4bef-b917-483a89219247-istio-podinfo\") pod \"e2e-noten-066ed8-openshift-default-547b48b74-ztbdw\" (UID: \"4524090d-1ce2-4bef-b917-483a89219247\") " pod="openshift-ingress/e2e-noten-066ed8-openshift-default-547b48b74-ztbdw" Sep 10 15:58:56.133564 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:56.133536 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5lmr\" (UniqueName: \"kubernetes.io/projected/4524090d-1ce2-4bef-b917-483a89219247-kube-api-access-p5lmr\") pod \"e2e-noten-066ed8-openshift-default-547b48b74-ztbdw\" (UID: \"4524090d-1ce2-4bef-b917-483a89219247\") " pod="openshift-ingress/e2e-noten-066ed8-openshift-default-547b48b74-ztbdw" Sep 10 15:58:56.133659 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:56.133540 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4524090d-1ce2-4bef-b917-483a89219247-istio-token\") pod \"e2e-noten-066ed8-openshift-default-547b48b74-ztbdw\" (UID: \"4524090d-1ce2-4bef-b917-483a89219247\") " pod="openshift-ingress/e2e-noten-066ed8-openshift-default-547b48b74-ztbdw" Sep 10 15:58:56.140433 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:56.140391 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:56.226455 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:56.226390 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-066ed8-openshift-default-547b48b74-ztbdw" Sep 10 15:58:56.530782 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:56.530757 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-066ed8-openshift-default-547b48b74-ztbdw"] Sep 10 15:58:56.532820 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:58:56.532796 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4524090d_1ce2_4bef_b917_483a89219247.slice/crio-27301076bb588044aa2e232c4e3e3e12c8c891ff89d9a86b45217c52765d320d WatchSource:0}: Error finding container 27301076bb588044aa2e232c4e3e3e12c8c891ff89d9a86b45217c52765d320d: Status 404 returned error can't find the container with id 27301076bb588044aa2e232c4e3e3e12c8c891ff89d9a86b45217c52765d320d Sep 10 15:58:56.535373 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:56.535341 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 15:58:56.535452 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:56.535408 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 15:58:56.535452 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:56.535437 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 15:58:57.207295 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:57.207266 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-066ed8-openshift-default-547b48b74-ztbdw" event={"ID":"4524090d-1ce2-4bef-b917-483a89219247","Type":"ContainerStarted","Data":"0a99e528629b4c0a3e8e5888df0cde3fd2b7ffb83e225f2376e273eb82537172"} Sep 10 15:58:57.207679 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:57.207320 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-066ed8-openshift-default-547b48b74-ztbdw" event={"ID":"4524090d-1ce2-4bef-b917-483a89219247","Type":"ContainerStarted","Data":"27301076bb588044aa2e232c4e3e3e12c8c891ff89d9a86b45217c52765d320d"} Sep 10 15:58:57.226802 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:57.226776 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-066ed8-openshift-default-547b48b74-ztbdw" Sep 10 15:58:57.231309 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:57.231265 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-066ed8-openshift-default-547b48b74-ztbdw" podStartSLOduration=2.231250551 podStartE2EDuration="2.231250551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:58:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:58:57.229588238 +0000 UTC m=+1014.583107420" watchObservedRunningTime="2026-09-10 15:58:57.231250551 +0000 UTC m=+1014.584769713" Sep 10 15:58:57.232344 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:57.232318 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-066ed8-openshift-default-547b48b74-ztbdw" Sep 10 15:58:57.530023 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:57.529944 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:57.639879 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:57.639847 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:57.790923 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:57.790894 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:58.213170 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:58.213088 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-066ed8-openshift-default-547b48b74-ztbdw" Sep 10 15:58:58.214002 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:58.213980 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-066ed8-openshift-default-547b48b74-ztbdw" Sep 10 15:58:58.592173 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:58.592136 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:58.712111 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:58.712083 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:58.801128 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:58:58.801096 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:01.103959 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:01.103918 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-77585476db-g2dkb" podUID="7c7adc7e-cee3-41a0-9c55-61f882ff454a" containerName="console" containerID="cri-o://fc0cec40f665b2a5c6bbd550309cec757f57e7aebfdaba8bd33efa6094b6a5c7" gracePeriod=15 Sep 10 15:59:01.230349 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:01.230322 2576 generic.go:356] "Generic (PLEG): container finished" podID="7c7adc7e-cee3-41a0-9c55-61f882ff454a" containerID="fc0cec40f665b2a5c6bbd550309cec757f57e7aebfdaba8bd33efa6094b6a5c7" exitCode=0 Sep 10 15:59:01.230459 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:01.230395 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77585476db-g2dkb" event={"ID":"7c7adc7e-cee3-41a0-9c55-61f882ff454a","Type":"ContainerDied","Data":"fc0cec40f665b2a5c6bbd550309cec757f57e7aebfdaba8bd33efa6094b6a5c7"} Sep 10 15:59:01.364011 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:01.363953 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77585476db-g2dkb" Sep 10 15:59:01.476221 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:01.476190 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7c7adc7e-cee3-41a0-9c55-61f882ff454a-service-ca\") pod \"7c7adc7e-cee3-41a0-9c55-61f882ff454a\" (UID: \"7c7adc7e-cee3-41a0-9c55-61f882ff454a\") " Sep 10 15:59:01.476384 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:01.476243 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c7adc7e-cee3-41a0-9c55-61f882ff454a-trusted-ca-bundle\") pod \"7c7adc7e-cee3-41a0-9c55-61f882ff454a\" (UID: \"7c7adc7e-cee3-41a0-9c55-61f882ff454a\") " Sep 10 15:59:01.476384 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:01.476277 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7c7adc7e-cee3-41a0-9c55-61f882ff454a-console-oauth-config\") pod \"7c7adc7e-cee3-41a0-9c55-61f882ff454a\" (UID: \"7c7adc7e-cee3-41a0-9c55-61f882ff454a\") " Sep 10 15:59:01.476384 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:01.476308 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v24hn\" (UniqueName: \"kubernetes.io/projected/7c7adc7e-cee3-41a0-9c55-61f882ff454a-kube-api-access-v24hn\") pod \"7c7adc7e-cee3-41a0-9c55-61f882ff454a\" (UID: \"7c7adc7e-cee3-41a0-9c55-61f882ff454a\") " Sep 10 15:59:01.476384 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:01.476363 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c7adc7e-cee3-41a0-9c55-61f882ff454a-console-serving-cert\") pod \"7c7adc7e-cee3-41a0-9c55-61f882ff454a\" (UID: \"7c7adc7e-cee3-41a0-9c55-61f882ff454a\") " Sep 10 15:59:01.476601 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:01.476437 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7c7adc7e-cee3-41a0-9c55-61f882ff454a-oauth-serving-cert\") pod \"7c7adc7e-cee3-41a0-9c55-61f882ff454a\" (UID: \"7c7adc7e-cee3-41a0-9c55-61f882ff454a\") " Sep 10 15:59:01.476601 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:01.476466 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7c7adc7e-cee3-41a0-9c55-61f882ff454a-console-config\") pod \"7c7adc7e-cee3-41a0-9c55-61f882ff454a\" (UID: \"7c7adc7e-cee3-41a0-9c55-61f882ff454a\") " Sep 10 15:59:01.476705 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:01.476610 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c7adc7e-cee3-41a0-9c55-61f882ff454a-service-ca" (OuterVolumeSpecName: "service-ca") pod "7c7adc7e-cee3-41a0-9c55-61f882ff454a" (UID: "7c7adc7e-cee3-41a0-9c55-61f882ff454a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:59:01.476871 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:01.476840 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c7adc7e-cee3-41a0-9c55-61f882ff454a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7c7adc7e-cee3-41a0-9c55-61f882ff454a" (UID: "7c7adc7e-cee3-41a0-9c55-61f882ff454a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:59:01.477034 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:01.476885 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c7adc7e-cee3-41a0-9c55-61f882ff454a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7c7adc7e-cee3-41a0-9c55-61f882ff454a" (UID: "7c7adc7e-cee3-41a0-9c55-61f882ff454a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:59:01.477034 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:01.476969 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c7adc7e-cee3-41a0-9c55-61f882ff454a-console-config" (OuterVolumeSpecName: "console-config") pod "7c7adc7e-cee3-41a0-9c55-61f882ff454a" (UID: "7c7adc7e-cee3-41a0-9c55-61f882ff454a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:59:01.477034 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:01.476992 2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7c7adc7e-cee3-41a0-9c55-61f882ff454a-service-ca\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:59:01.477360 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:01.477136 2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c7adc7e-cee3-41a0-9c55-61f882ff454a-trusted-ca-bundle\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:59:01.478605 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:01.478578 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c7adc7e-cee3-41a0-9c55-61f882ff454a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7c7adc7e-cee3-41a0-9c55-61f882ff454a" (UID: "7c7adc7e-cee3-41a0-9c55-61f882ff454a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:59:01.478692 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:01.478607 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c7adc7e-cee3-41a0-9c55-61f882ff454a-kube-api-access-v24hn" (OuterVolumeSpecName: "kube-api-access-v24hn") pod "7c7adc7e-cee3-41a0-9c55-61f882ff454a" (UID: "7c7adc7e-cee3-41a0-9c55-61f882ff454a"). InnerVolumeSpecName "kube-api-access-v24hn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:59:01.478692 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:01.478606 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c7adc7e-cee3-41a0-9c55-61f882ff454a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7c7adc7e-cee3-41a0-9c55-61f882ff454a" (UID: "7c7adc7e-cee3-41a0-9c55-61f882ff454a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:59:01.578419 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:01.578392 2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7c7adc7e-cee3-41a0-9c55-61f882ff454a-oauth-serving-cert\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:59:01.578419 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:01.578415 2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7c7adc7e-cee3-41a0-9c55-61f882ff454a-console-config\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:59:01.578563 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:01.578424 2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7c7adc7e-cee3-41a0-9c55-61f882ff454a-console-oauth-config\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:59:01.578563 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:01.578435 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v24hn\" (UniqueName: \"kubernetes.io/projected/7c7adc7e-cee3-41a0-9c55-61f882ff454a-kube-api-access-v24hn\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:59:01.578563 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:01.578444 2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c7adc7e-cee3-41a0-9c55-61f882ff454a-console-serving-cert\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:59:02.237254 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:02.237227 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77585476db-g2dkb" Sep 10 15:59:02.237663 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:02.237228 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77585476db-g2dkb" event={"ID":"7c7adc7e-cee3-41a0-9c55-61f882ff454a","Type":"ContainerDied","Data":"87e83dbfd4cceba5f2692d0e5712d33801c8dc82808cabba514027e223730e1a"} Sep 10 15:59:02.237663 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:02.237360 2576 scope.go:117] "RemoveContainer" containerID="fc0cec40f665b2a5c6bbd550309cec757f57e7aebfdaba8bd33efa6094b6a5c7" Sep 10 15:59:02.261114 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:02.261083 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77585476db-g2dkb"] Sep 10 15:59:02.266189 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:02.266166 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-77585476db-g2dkb"] Sep 10 15:59:03.169378 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:03.169342 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c7adc7e-cee3-41a0-9c55-61f882ff454a" path="/var/lib/kubelet/pods/7c7adc7e-cee3-41a0-9c55-61f882ff454a/volumes" Sep 10 15:59:05.055313 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:05.055274 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:09.340155 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:09.340112 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:10.127709 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:10.127674 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:10.389460 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:10.389386 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 15:59:16.341270 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:16.341238 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:16.449790 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:16.449764 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:16.573161 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:16.573129 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:16.686820 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:16.686722 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:17.106368 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:17.106335 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:17.373542 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:17.373452 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:17.495576 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:17.495536 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:17.654061 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:17.653963 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:18.016465 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:18.016374 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:18.089707 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:18.089676 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-066ed8-openshift-default-547b48b74-ztbdw"] Sep 10 15:59:18.089932 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:18.089911 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-066ed8-openshift-default-547b48b74-ztbdw" podUID="4524090d-1ce2-4bef-b917-483a89219247" containerName="istio-proxy" containerID="cri-o://0a99e528629b4c0a3e8e5888df0cde3fd2b7ffb83e225f2376e273eb82537172" gracePeriod=30 Sep 10 15:59:18.135058 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:18.135026 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:18.327252 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:18.327215 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:18.427278 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:18.427245 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:19.106197 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:19.106161 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:19.248353 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:19.248313 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:19.464661 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:19.464572 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:22.287333 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:22.287294 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 15:59:23.351960 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:23.351884 2576 generic.go:356] "Generic (PLEG): container finished" podID="4524090d-1ce2-4bef-b917-483a89219247" containerID="0a99e528629b4c0a3e8e5888df0cde3fd2b7ffb83e225f2376e273eb82537172" exitCode=0 Sep 10 15:59:23.351960 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:23.351944 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-066ed8-openshift-default-547b48b74-ztbdw" event={"ID":"4524090d-1ce2-4bef-b917-483a89219247","Type":"ContainerDied","Data":"0a99e528629b4c0a3e8e5888df0cde3fd2b7ffb83e225f2376e273eb82537172"} Sep 10 15:59:23.352319 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:23.351985 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-066ed8-openshift-default-547b48b74-ztbdw" event={"ID":"4524090d-1ce2-4bef-b917-483a89219247","Type":"ContainerDied","Data":"27301076bb588044aa2e232c4e3e3e12c8c891ff89d9a86b45217c52765d320d"} Sep 10 15:59:23.352319 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:23.352008 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27301076bb588044aa2e232c4e3e3e12c8c891ff89d9a86b45217c52765d320d" Sep 10 15:59:23.362373 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:23.362352 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-066ed8-openshift-default-547b48b74-ztbdw" Sep 10 15:59:23.462244 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:23.462213 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4524090d-1ce2-4bef-b917-483a89219247-workload-socket\") pod \"4524090d-1ce2-4bef-b917-483a89219247\" (UID: \"4524090d-1ce2-4bef-b917-483a89219247\") " Sep 10 15:59:23.462244 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:23.462242 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4524090d-1ce2-4bef-b917-483a89219247-istio-podinfo\") pod \"4524090d-1ce2-4bef-b917-483a89219247\" (UID: \"4524090d-1ce2-4bef-b917-483a89219247\") " Sep 10 15:59:23.462477 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:23.462273 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4524090d-1ce2-4bef-b917-483a89219247-istio-token\") pod \"4524090d-1ce2-4bef-b917-483a89219247\" (UID: \"4524090d-1ce2-4bef-b917-483a89219247\") " Sep 10 15:59:23.462477 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:23.462295 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4524090d-1ce2-4bef-b917-483a89219247-istio-envoy\") pod \"4524090d-1ce2-4bef-b917-483a89219247\" (UID: \"4524090d-1ce2-4bef-b917-483a89219247\") " Sep 10 15:59:23.462477 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:23.462313 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4524090d-1ce2-4bef-b917-483a89219247-credential-socket\") pod \"4524090d-1ce2-4bef-b917-483a89219247\" (UID: \"4524090d-1ce2-4bef-b917-483a89219247\") " Sep 10 15:59:23.462477 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:23.462353 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p5lmr\" (UniqueName: \"kubernetes.io/projected/4524090d-1ce2-4bef-b917-483a89219247-kube-api-access-p5lmr\") pod \"4524090d-1ce2-4bef-b917-483a89219247\" (UID: \"4524090d-1ce2-4bef-b917-483a89219247\") " Sep 10 15:59:23.462477 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:23.462382 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4524090d-1ce2-4bef-b917-483a89219247-istio-data\") pod \"4524090d-1ce2-4bef-b917-483a89219247\" (UID: \"4524090d-1ce2-4bef-b917-483a89219247\") " Sep 10 15:59:23.462477 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:23.462421 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4524090d-1ce2-4bef-b917-483a89219247-istiod-ca-cert\") pod \"4524090d-1ce2-4bef-b917-483a89219247\" (UID: \"4524090d-1ce2-4bef-b917-483a89219247\") " Sep 10 15:59:23.462477 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:23.462463 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4524090d-1ce2-4bef-b917-483a89219247-workload-certs\") pod \"4524090d-1ce2-4bef-b917-483a89219247\" (UID: \"4524090d-1ce2-4bef-b917-483a89219247\") " Sep 10 15:59:23.462875 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:23.462547 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4524090d-1ce2-4bef-b917-483a89219247-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4524090d-1ce2-4bef-b917-483a89219247" (UID: "4524090d-1ce2-4bef-b917-483a89219247"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:59:23.462932 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:23.462888 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4524090d-1ce2-4bef-b917-483a89219247-workload-socket\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:59:23.462991 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:23.462926 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4524090d-1ce2-4bef-b917-483a89219247-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4524090d-1ce2-4bef-b917-483a89219247" (UID: "4524090d-1ce2-4bef-b917-483a89219247"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:59:23.463096 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:23.463073 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4524090d-1ce2-4bef-b917-483a89219247-istio-data" (OuterVolumeSpecName: "istio-data") pod "4524090d-1ce2-4bef-b917-483a89219247" (UID: "4524090d-1ce2-4bef-b917-483a89219247"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:59:23.463096 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:23.463082 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4524090d-1ce2-4bef-b917-483a89219247-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4524090d-1ce2-4bef-b917-483a89219247" (UID: "4524090d-1ce2-4bef-b917-483a89219247"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:59:23.463382 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:23.463143 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4524090d-1ce2-4bef-b917-483a89219247-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4524090d-1ce2-4bef-b917-483a89219247" (UID: "4524090d-1ce2-4bef-b917-483a89219247"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:59:23.464796 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:23.464748 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4524090d-1ce2-4bef-b917-483a89219247-istio-token" (OuterVolumeSpecName: "istio-token") pod "4524090d-1ce2-4bef-b917-483a89219247" (UID: "4524090d-1ce2-4bef-b917-483a89219247"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:59:23.465036 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:23.465015 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4524090d-1ce2-4bef-b917-483a89219247-kube-api-access-p5lmr" (OuterVolumeSpecName: "kube-api-access-p5lmr") pod "4524090d-1ce2-4bef-b917-483a89219247" (UID: "4524090d-1ce2-4bef-b917-483a89219247"). InnerVolumeSpecName "kube-api-access-p5lmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:59:23.465149 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:23.465134 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4524090d-1ce2-4bef-b917-483a89219247-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4524090d-1ce2-4bef-b917-483a89219247" (UID: "4524090d-1ce2-4bef-b917-483a89219247"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:59:23.465241 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:23.465221 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4524090d-1ce2-4bef-b917-483a89219247-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4524090d-1ce2-4bef-b917-483a89219247" (UID: "4524090d-1ce2-4bef-b917-483a89219247"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 15:59:23.479773 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:23.479717 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:23.563308 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:23.563287 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4524090d-1ce2-4bef-b917-483a89219247-istio-token\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:59:23.563308 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:23.563306 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4524090d-1ce2-4bef-b917-483a89219247-istio-envoy\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:59:23.563437 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:23.563315 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4524090d-1ce2-4bef-b917-483a89219247-credential-socket\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:59:23.563437 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:23.563323 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5lmr\" (UniqueName: \"kubernetes.io/projected/4524090d-1ce2-4bef-b917-483a89219247-kube-api-access-p5lmr\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:59:23.563437 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:23.563332 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4524090d-1ce2-4bef-b917-483a89219247-istio-data\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:59:23.563437 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:23.563340 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4524090d-1ce2-4bef-b917-483a89219247-istiod-ca-cert\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:59:23.563437 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:23.563347 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4524090d-1ce2-4bef-b917-483a89219247-workload-certs\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:59:23.563437 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:23.563355 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4524090d-1ce2-4bef-b917-483a89219247-istio-podinfo\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:59:24.357477 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:24.357444 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-066ed8-openshift-default-547b48b74-ztbdw" Sep 10 15:59:24.383242 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:24.383211 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-066ed8-openshift-default-547b48b74-ztbdw"] Sep 10 15:59:24.391931 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:24.391908 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-066ed8-openshift-default-547b48b74-ztbdw"] Sep 10 15:59:25.168792 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:25.168762 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4524090d-1ce2-4bef-b917-483a89219247" path="/var/lib/kubelet/pods/4524090d-1ce2-4bef-b917-483a89219247/volumes" Sep 10 15:59:25.385864 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:25.385835 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:25.476513 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:25.476424 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:26.079682 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:26.079648 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:29.794221 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:29.794178 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:29.829684 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:29.829654 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-789be7-99b575b9c-2ww9p"] Sep 10 15:59:29.829933 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:29.829905 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-789be7-99b575b9c-2ww9p" podUID="902985a8-c8fb-4afe-8a77-efb13f89f692" containerName="payload-processing" containerID="cri-o://508d4707ec418d22999303ff37b42058c873e066cd5fb3825905f79bfc47cc9b" gracePeriod=30 Sep 10 15:59:29.840232 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:29.840204 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-789be7-6cddb9575d-7m8jc"] Sep 10 15:59:29.840476 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:29.840444 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-789be7-6cddb9575d-7m8jc" podUID="66c15852-b623-45f9-9289-abafb9ef4329" containerName="payload-pre-processing" containerID="cri-o://5cd8d47f37e0ac014698331593ce12eaa248263cf255206806ab76213da7f96b" gracePeriod=30 Sep 10 15:59:29.875567 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:29.875513 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:29.977272 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:29.977177 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:30.067304 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:30.067274 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:30.087179 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:30.087156 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-789be7-99b575b9c-2ww9p" Sep 10 15:59:30.123928 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:30.123903 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/902985a8-c8fb-4afe-8a77-efb13f89f692-plugins-config-volume\") pod \"902985a8-c8fb-4afe-8a77-efb13f89f692\" (UID: \"902985a8-c8fb-4afe-8a77-efb13f89f692\") " Sep 10 15:59:30.124052 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:30.123954 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qffzz\" (UniqueName: \"kubernetes.io/projected/902985a8-c8fb-4afe-8a77-efb13f89f692-kube-api-access-qffzz\") pod \"902985a8-c8fb-4afe-8a77-efb13f89f692\" (UID: \"902985a8-c8fb-4afe-8a77-efb13f89f692\") " Sep 10 15:59:30.124262 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:30.124233 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/902985a8-c8fb-4afe-8a77-efb13f89f692-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "902985a8-c8fb-4afe-8a77-efb13f89f692" (UID: "902985a8-c8fb-4afe-8a77-efb13f89f692"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:59:30.124780 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:30.124761 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-789be7-6cddb9575d-7m8jc" Sep 10 15:59:30.126064 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:30.126035 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/902985a8-c8fb-4afe-8a77-efb13f89f692-kube-api-access-qffzz" (OuterVolumeSpecName: "kube-api-access-qffzz") pod "902985a8-c8fb-4afe-8a77-efb13f89f692" (UID: "902985a8-c8fb-4afe-8a77-efb13f89f692"). InnerVolumeSpecName "kube-api-access-qffzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:59:30.224776 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:30.224719 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cbvfh\" (UniqueName: \"kubernetes.io/projected/66c15852-b623-45f9-9289-abafb9ef4329-kube-api-access-cbvfh\") pod \"66c15852-b623-45f9-9289-abafb9ef4329\" (UID: \"66c15852-b623-45f9-9289-abafb9ef4329\") " Sep 10 15:59:30.224942 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:30.224812 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66c15852-b623-45f9-9289-abafb9ef4329-plugins-config-volume\") pod \"66c15852-b623-45f9-9289-abafb9ef4329\" (UID: \"66c15852-b623-45f9-9289-abafb9ef4329\") " Sep 10 15:59:30.225127 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:30.225106 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/902985a8-c8fb-4afe-8a77-efb13f89f692-plugins-config-volume\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:59:30.225199 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:30.225122 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66c15852-b623-45f9-9289-abafb9ef4329-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "66c15852-b623-45f9-9289-abafb9ef4329" (UID: "66c15852-b623-45f9-9289-abafb9ef4329"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:59:30.225199 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:30.225133 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qffzz\" (UniqueName: \"kubernetes.io/projected/902985a8-c8fb-4afe-8a77-efb13f89f692-kube-api-access-qffzz\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:59:30.227115 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:30.227088 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66c15852-b623-45f9-9289-abafb9ef4329-kube-api-access-cbvfh" (OuterVolumeSpecName: "kube-api-access-cbvfh") pod "66c15852-b623-45f9-9289-abafb9ef4329" (UID: "66c15852-b623-45f9-9289-abafb9ef4329"). InnerVolumeSpecName "kube-api-access-cbvfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:59:30.325649 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:30.325580 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbvfh\" (UniqueName: \"kubernetes.io/projected/66c15852-b623-45f9-9289-abafb9ef4329-kube-api-access-cbvfh\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:59:30.325649 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:30.325607 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66c15852-b623-45f9-9289-abafb9ef4329-plugins-config-volume\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:59:30.388167 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:30.388132 2576 generic.go:356] "Generic (PLEG): container finished" podID="902985a8-c8fb-4afe-8a77-efb13f89f692" containerID="508d4707ec418d22999303ff37b42058c873e066cd5fb3825905f79bfc47cc9b" exitCode=0 Sep 10 15:59:30.388315 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:30.388179 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-789be7-99b575b9c-2ww9p" Sep 10 15:59:30.388315 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:30.388212 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-789be7-99b575b9c-2ww9p" event={"ID":"902985a8-c8fb-4afe-8a77-efb13f89f692","Type":"ContainerDied","Data":"508d4707ec418d22999303ff37b42058c873e066cd5fb3825905f79bfc47cc9b"} Sep 10 15:59:30.388315 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:30.388271 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-789be7-99b575b9c-2ww9p" event={"ID":"902985a8-c8fb-4afe-8a77-efb13f89f692","Type":"ContainerDied","Data":"53e1efcb25ee25e94604e580b1e96a0fdaec13d63f115b9a3b943c02475b7666"} Sep 10 15:59:30.388315 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:30.388293 2576 scope.go:117] "RemoveContainer" containerID="508d4707ec418d22999303ff37b42058c873e066cd5fb3825905f79bfc47cc9b" Sep 10 15:59:30.389648 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:30.389625 2576 generic.go:356] "Generic (PLEG): container finished" podID="66c15852-b623-45f9-9289-abafb9ef4329" containerID="5cd8d47f37e0ac014698331593ce12eaa248263cf255206806ab76213da7f96b" exitCode=0 Sep 10 15:59:30.389799 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:30.389669 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-789be7-6cddb9575d-7m8jc" Sep 10 15:59:30.389799 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:30.389688 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-789be7-6cddb9575d-7m8jc" event={"ID":"66c15852-b623-45f9-9289-abafb9ef4329","Type":"ContainerDied","Data":"5cd8d47f37e0ac014698331593ce12eaa248263cf255206806ab76213da7f96b"} Sep 10 15:59:30.389799 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:30.389726 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-789be7-6cddb9575d-7m8jc" event={"ID":"66c15852-b623-45f9-9289-abafb9ef4329","Type":"ContainerDied","Data":"d9065f2329c7af978472d9cd6bc2db60f4b6eb4f4f634e650f09b90a467949d5"} Sep 10 15:59:30.397930 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:30.397907 2576 scope.go:117] "RemoveContainer" containerID="508d4707ec418d22999303ff37b42058c873e066cd5fb3825905f79bfc47cc9b" Sep 10 15:59:30.398225 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:59:30.398203 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"508d4707ec418d22999303ff37b42058c873e066cd5fb3825905f79bfc47cc9b\": container with ID starting with 508d4707ec418d22999303ff37b42058c873e066cd5fb3825905f79bfc47cc9b not found: ID does not exist" containerID="508d4707ec418d22999303ff37b42058c873e066cd5fb3825905f79bfc47cc9b" Sep 10 15:59:30.398316 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:30.398234 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"508d4707ec418d22999303ff37b42058c873e066cd5fb3825905f79bfc47cc9b"} err="failed to get container status \"508d4707ec418d22999303ff37b42058c873e066cd5fb3825905f79bfc47cc9b\": rpc error: code = NotFound desc = could not find container \"508d4707ec418d22999303ff37b42058c873e066cd5fb3825905f79bfc47cc9b\": container with ID starting with 508d4707ec418d22999303ff37b42058c873e066cd5fb3825905f79bfc47cc9b not found: ID does not exist" Sep 10 15:59:30.398316 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:30.398256 2576 scope.go:117] "RemoveContainer" containerID="5cd8d47f37e0ac014698331593ce12eaa248263cf255206806ab76213da7f96b" Sep 10 15:59:30.406852 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:30.406832 2576 scope.go:117] "RemoveContainer" containerID="5cd8d47f37e0ac014698331593ce12eaa248263cf255206806ab76213da7f96b" Sep 10 15:59:30.407116 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:59:30.407097 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cd8d47f37e0ac014698331593ce12eaa248263cf255206806ab76213da7f96b\": container with ID starting with 5cd8d47f37e0ac014698331593ce12eaa248263cf255206806ab76213da7f96b not found: ID does not exist" containerID="5cd8d47f37e0ac014698331593ce12eaa248263cf255206806ab76213da7f96b" Sep 10 15:59:30.407182 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:30.407122 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cd8d47f37e0ac014698331593ce12eaa248263cf255206806ab76213da7f96b"} err="failed to get container status \"5cd8d47f37e0ac014698331593ce12eaa248263cf255206806ab76213da7f96b\": rpc error: code = NotFound desc = could not find container \"5cd8d47f37e0ac014698331593ce12eaa248263cf255206806ab76213da7f96b\": container with ID starting with 5cd8d47f37e0ac014698331593ce12eaa248263cf255206806ab76213da7f96b not found: ID does not exist" Sep 10 15:59:30.414713 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:30.414694 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-789be7-99b575b9c-2ww9p"] Sep 10 15:59:30.425828 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:30.425756 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:30.427024 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:30.427006 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-789be7-99b575b9c-2ww9p"] Sep 10 15:59:30.439489 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:30.439470 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-789be7-6cddb9575d-7m8jc"] Sep 10 15:59:30.444580 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:30.444562 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-789be7-6cddb9575d-7m8jc"] Sep 10 15:59:31.169236 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:31.169154 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66c15852-b623-45f9-9289-abafb9ef4329" path="/var/lib/kubelet/pods/66c15852-b623-45f9-9289-abafb9ef4329/volumes" Sep 10 15:59:31.169573 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:31.169496 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="902985a8-c8fb-4afe-8a77-efb13f89f692" path="/var/lib/kubelet/pods/902985a8-c8fb-4afe-8a77-efb13f89f692/volumes" Sep 10 15:59:32.686958 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:32.686921 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:32.793569 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:32.793527 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:32.940375 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:32.940280 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:33.244036 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:33.243949 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj"] Sep 10 15:59:33.244373 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:33.244356 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4524090d-1ce2-4bef-b917-483a89219247" containerName="istio-proxy" Sep 10 15:59:33.244373 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:33.244375 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4524090d-1ce2-4bef-b917-483a89219247" containerName="istio-proxy" Sep 10 15:59:33.244507 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:33.244385 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="902985a8-c8fb-4afe-8a77-efb13f89f692" containerName="payload-processing" Sep 10 15:59:33.244507 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:33.244391 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="902985a8-c8fb-4afe-8a77-efb13f89f692" containerName="payload-processing" Sep 10 15:59:33.244507 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:33.244404 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c7adc7e-cee3-41a0-9c55-61f882ff454a" containerName="console" Sep 10 15:59:33.244507 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:33.244409 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c7adc7e-cee3-41a0-9c55-61f882ff454a" containerName="console" Sep 10 15:59:33.244507 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:33.244416 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66c15852-b623-45f9-9289-abafb9ef4329" containerName="payload-pre-processing" Sep 10 15:59:33.244507 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:33.244422 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="66c15852-b623-45f9-9289-abafb9ef4329" containerName="payload-pre-processing" Sep 10 15:59:33.244507 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:33.244486 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4524090d-1ce2-4bef-b917-483a89219247" containerName="istio-proxy" Sep 10 15:59:33.244507 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:33.244500 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="66c15852-b623-45f9-9289-abafb9ef4329" containerName="payload-pre-processing" Sep 10 15:59:33.244507 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:33.244510 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="902985a8-c8fb-4afe-8a77-efb13f89f692" containerName="payload-processing" Sep 10 15:59:33.244837 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:33.244520 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c7adc7e-cee3-41a0-9c55-61f882ff454a" containerName="console" Sep 10 15:59:33.249045 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:33.249028 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj" Sep 10 15:59:33.253853 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:33.253833 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-678f625f-openshift-default-dockercfg-cfq8b\"" Sep 10 15:59:33.270609 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:33.270577 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj"] Sep 10 15:59:33.287369 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:33.287338 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:33.351313 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:33.350919 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d500885-c94a-4f5e-b9da-61e310e6ad7c-workload-socket\") pod \"e2e-mt-678f625f-openshift-default-69b54854cb-sxspj\" (UID: \"1d500885-c94a-4f5e-b9da-61e310e6ad7c\") " pod="openshift-ingress/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj" Sep 10 15:59:33.351313 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:33.350971 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d500885-c94a-4f5e-b9da-61e310e6ad7c-credential-socket\") pod \"e2e-mt-678f625f-openshift-default-69b54854cb-sxspj\" (UID: \"1d500885-c94a-4f5e-b9da-61e310e6ad7c\") " pod="openshift-ingress/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj" Sep 10 15:59:33.351313 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:33.351003 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnsfs\" (UniqueName: \"kubernetes.io/projected/1d500885-c94a-4f5e-b9da-61e310e6ad7c-kube-api-access-fnsfs\") pod \"e2e-mt-678f625f-openshift-default-69b54854cb-sxspj\" (UID: \"1d500885-c94a-4f5e-b9da-61e310e6ad7c\") " pod="openshift-ingress/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj" Sep 10 15:59:33.351313 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:33.351094 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d500885-c94a-4f5e-b9da-61e310e6ad7c-workload-certs\") pod \"e2e-mt-678f625f-openshift-default-69b54854cb-sxspj\" (UID: \"1d500885-c94a-4f5e-b9da-61e310e6ad7c\") " pod="openshift-ingress/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj" Sep 10 15:59:33.351313 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:33.351140 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d500885-c94a-4f5e-b9da-61e310e6ad7c-istio-envoy\") pod \"e2e-mt-678f625f-openshift-default-69b54854cb-sxspj\" (UID: \"1d500885-c94a-4f5e-b9da-61e310e6ad7c\") " pod="openshift-ingress/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj" Sep 10 15:59:33.351313 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:33.351165 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d500885-c94a-4f5e-b9da-61e310e6ad7c-istio-podinfo\") pod \"e2e-mt-678f625f-openshift-default-69b54854cb-sxspj\" (UID: \"1d500885-c94a-4f5e-b9da-61e310e6ad7c\") " pod="openshift-ingress/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj" Sep 10 15:59:33.351313 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:33.351227 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d500885-c94a-4f5e-b9da-61e310e6ad7c-istiod-ca-cert\") pod \"e2e-mt-678f625f-openshift-default-69b54854cb-sxspj\" (UID: \"1d500885-c94a-4f5e-b9da-61e310e6ad7c\") " pod="openshift-ingress/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj" Sep 10 15:59:33.351313 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:33.351281 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d500885-c94a-4f5e-b9da-61e310e6ad7c-istio-token\") pod \"e2e-mt-678f625f-openshift-default-69b54854cb-sxspj\" (UID: \"1d500885-c94a-4f5e-b9da-61e310e6ad7c\") " pod="openshift-ingress/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj" Sep 10 15:59:33.351313 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:33.351317 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d500885-c94a-4f5e-b9da-61e310e6ad7c-istio-data\") pod \"e2e-mt-678f625f-openshift-default-69b54854cb-sxspj\" (UID: \"1d500885-c94a-4f5e-b9da-61e310e6ad7c\") " pod="openshift-ingress/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj" Sep 10 15:59:33.410713 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:33.410672 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:33.452717 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:33.452679 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d500885-c94a-4f5e-b9da-61e310e6ad7c-istio-data\") pod \"e2e-mt-678f625f-openshift-default-69b54854cb-sxspj\" (UID: \"1d500885-c94a-4f5e-b9da-61e310e6ad7c\") " pod="openshift-ingress/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj" Sep 10 15:59:33.452936 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:33.452775 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d500885-c94a-4f5e-b9da-61e310e6ad7c-workload-socket\") pod \"e2e-mt-678f625f-openshift-default-69b54854cb-sxspj\" (UID: \"1d500885-c94a-4f5e-b9da-61e310e6ad7c\") " pod="openshift-ingress/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj" Sep 10 15:59:33.452936 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:33.452801 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d500885-c94a-4f5e-b9da-61e310e6ad7c-credential-socket\") pod \"e2e-mt-678f625f-openshift-default-69b54854cb-sxspj\" (UID: \"1d500885-c94a-4f5e-b9da-61e310e6ad7c\") " pod="openshift-ingress/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj" Sep 10 15:59:33.452936 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:33.452829 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnsfs\" (UniqueName: \"kubernetes.io/projected/1d500885-c94a-4f5e-b9da-61e310e6ad7c-kube-api-access-fnsfs\") pod \"e2e-mt-678f625f-openshift-default-69b54854cb-sxspj\" (UID: \"1d500885-c94a-4f5e-b9da-61e310e6ad7c\") " pod="openshift-ingress/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj" Sep 10 15:59:33.452936 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:33.452909 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d500885-c94a-4f5e-b9da-61e310e6ad7c-workload-certs\") pod \"e2e-mt-678f625f-openshift-default-69b54854cb-sxspj\" (UID: \"1d500885-c94a-4f5e-b9da-61e310e6ad7c\") " pod="openshift-ingress/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj" Sep 10 15:59:33.453157 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:33.452947 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d500885-c94a-4f5e-b9da-61e310e6ad7c-istio-envoy\") pod \"e2e-mt-678f625f-openshift-default-69b54854cb-sxspj\" (UID: \"1d500885-c94a-4f5e-b9da-61e310e6ad7c\") " pod="openshift-ingress/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj" Sep 10 15:59:33.453157 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:33.452971 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d500885-c94a-4f5e-b9da-61e310e6ad7c-istio-podinfo\") pod \"e2e-mt-678f625f-openshift-default-69b54854cb-sxspj\" (UID: \"1d500885-c94a-4f5e-b9da-61e310e6ad7c\") " pod="openshift-ingress/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj" Sep 10 15:59:33.453157 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:33.453034 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d500885-c94a-4f5e-b9da-61e310e6ad7c-istiod-ca-cert\") pod \"e2e-mt-678f625f-openshift-default-69b54854cb-sxspj\" (UID: \"1d500885-c94a-4f5e-b9da-61e310e6ad7c\") " pod="openshift-ingress/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj" Sep 10 15:59:33.453157 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:33.453070 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d500885-c94a-4f5e-b9da-61e310e6ad7c-istio-token\") pod \"e2e-mt-678f625f-openshift-default-69b54854cb-sxspj\" (UID: \"1d500885-c94a-4f5e-b9da-61e310e6ad7c\") " pod="openshift-ingress/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj" Sep 10 15:59:33.453361 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:33.453151 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d500885-c94a-4f5e-b9da-61e310e6ad7c-istio-data\") pod \"e2e-mt-678f625f-openshift-default-69b54854cb-sxspj\" (UID: \"1d500885-c94a-4f5e-b9da-61e310e6ad7c\") " pod="openshift-ingress/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj" Sep 10 15:59:33.453361 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:33.453181 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d500885-c94a-4f5e-b9da-61e310e6ad7c-credential-socket\") pod \"e2e-mt-678f625f-openshift-default-69b54854cb-sxspj\" (UID: \"1d500885-c94a-4f5e-b9da-61e310e6ad7c\") " pod="openshift-ingress/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj" Sep 10 15:59:33.453361 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:33.453222 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d500885-c94a-4f5e-b9da-61e310e6ad7c-workload-socket\") pod \"e2e-mt-678f625f-openshift-default-69b54854cb-sxspj\" (UID: \"1d500885-c94a-4f5e-b9da-61e310e6ad7c\") " pod="openshift-ingress/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj" Sep 10 15:59:33.453498 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:33.453470 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d500885-c94a-4f5e-b9da-61e310e6ad7c-workload-certs\") pod \"e2e-mt-678f625f-openshift-default-69b54854cb-sxspj\" (UID: \"1d500885-c94a-4f5e-b9da-61e310e6ad7c\") " pod="openshift-ingress/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj" Sep 10 15:59:33.453827 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:33.453724 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d500885-c94a-4f5e-b9da-61e310e6ad7c-istiod-ca-cert\") pod \"e2e-mt-678f625f-openshift-default-69b54854cb-sxspj\" (UID: \"1d500885-c94a-4f5e-b9da-61e310e6ad7c\") " pod="openshift-ingress/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj" Sep 10 15:59:33.455421 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:33.455391 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d500885-c94a-4f5e-b9da-61e310e6ad7c-istio-envoy\") pod \"e2e-mt-678f625f-openshift-default-69b54854cb-sxspj\" (UID: \"1d500885-c94a-4f5e-b9da-61e310e6ad7c\") " pod="openshift-ingress/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj" Sep 10 15:59:33.455648 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:33.455627 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d500885-c94a-4f5e-b9da-61e310e6ad7c-istio-podinfo\") pod \"e2e-mt-678f625f-openshift-default-69b54854cb-sxspj\" (UID: \"1d500885-c94a-4f5e-b9da-61e310e6ad7c\") " pod="openshift-ingress/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj" Sep 10 15:59:33.465364 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:33.465339 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d500885-c94a-4f5e-b9da-61e310e6ad7c-istio-token\") pod \"e2e-mt-678f625f-openshift-default-69b54854cb-sxspj\" (UID: \"1d500885-c94a-4f5e-b9da-61e310e6ad7c\") " pod="openshift-ingress/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj" Sep 10 15:59:33.473483 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:33.473462 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnsfs\" (UniqueName: \"kubernetes.io/projected/1d500885-c94a-4f5e-b9da-61e310e6ad7c-kube-api-access-fnsfs\") pod \"e2e-mt-678f625f-openshift-default-69b54854cb-sxspj\" (UID: \"1d500885-c94a-4f5e-b9da-61e310e6ad7c\") " pod="openshift-ingress/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj" Sep 10 15:59:33.559111 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:33.559083 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj" Sep 10 15:59:33.843983 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:33.843960 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj"] Sep 10 15:59:33.844880 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:59:33.844857 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d500885_c94a_4f5e_b9da_61e310e6ad7c.slice/crio-2ee08553edb96113f418ff894a3b89229a26ca952859e18ef8b7164109a0715a WatchSource:0}: Error finding container 2ee08553edb96113f418ff894a3b89229a26ca952859e18ef8b7164109a0715a: Status 404 returned error can't find the container with id 2ee08553edb96113f418ff894a3b89229a26ca952859e18ef8b7164109a0715a Sep 10 15:59:33.847089 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:33.847054 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 15:59:33.847162 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:33.847119 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 15:59:33.847162 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:33.847148 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 15:59:33.981280 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:33.981238 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 15:59:34.419381 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:34.419345 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj" event={"ID":"1d500885-c94a-4f5e-b9da-61e310e6ad7c","Type":"ContainerStarted","Data":"3513b9c880370e635603a4f9290c86b5a151b96a9b05f94148a8b59cfe99707d"} Sep 10 15:59:34.419577 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:34.419409 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj" event={"ID":"1d500885-c94a-4f5e-b9da-61e310e6ad7c","Type":"ContainerStarted","Data":"2ee08553edb96113f418ff894a3b89229a26ca952859e18ef8b7164109a0715a"} Sep 10 15:59:34.445483 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:34.445431 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj" podStartSLOduration=1.445415793 podStartE2EDuration="1.445415793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:59:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:59:34.442665761 +0000 UTC m=+1051.796184933" watchObservedRunningTime="2026-09-10 15:59:34.445415793 +0000 UTC m=+1051.798934954" Sep 10 15:59:34.559815 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:34.559768 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj" Sep 10 15:59:34.561789 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:34.561757 2576 patch_prober.go:28] interesting pod/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Sep 10 15:59:34.561945 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:34.561822 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj" podUID="1d500885-c94a-4f5e-b9da-61e310e6ad7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Sep 10 15:59:35.563787 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:35.563726 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj" Sep 10 15:59:36.398018 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:36.397981 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:36.430063 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:36.430034 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj" Sep 10 15:59:36.431040 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:36.431016 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj" Sep 10 15:59:36.496636 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:36.496599 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:37.220950 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:37.220907 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:37.324055 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:37.324013 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:37.468339 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:37.468304 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-789be7-openshift-default-6669946986-bchwf"] Sep 10 15:59:37.468596 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:37.468565 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-789be7-openshift-default-6669946986-bchwf" podUID="d22ca135-fb1d-41c7-bba6-9109e30cfe61" containerName="istio-proxy" containerID="cri-o://d14c027af6ffee6e7bd5c45dca0e656b1bd8a899b84462474dbcf4e580f38213" gracePeriod=30 Sep 10 15:59:37.502157 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:37.502079 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:37.656505 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:37.656470 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:37.775635 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:37.775548 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:38.506541 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:38.506492 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-678f625f-dbcf47945-9c9kv"] Sep 10 15:59:38.510288 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:38.510266 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-678f625f-dbcf47945-9c9kv" Sep 10 15:59:38.513359 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:38.513339 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-678f625f\"" Sep 10 15:59:38.513460 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:38.513375 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-678f625f-dockercfg-b4dtk\"" Sep 10 15:59:38.531423 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:38.531402 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-678f625f-dbcf47945-9c9kv"] Sep 10 15:59:38.537913 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:38.537889 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-678f625f-59c7bc5874-trjv6"] Sep 10 15:59:38.540541 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:38.540506 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:38.541972 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:38.541953 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-678f625f-59c7bc5874-trjv6" Sep 10 15:59:38.555776 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:38.555726 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-678f625f-59c7bc5874-trjv6"] Sep 10 15:59:38.607983 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:38.607939 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhk7v\" (UniqueName: \"kubernetes.io/projected/9fac4a7c-3089-4b7b-a111-fd10fa79f4df-kube-api-access-mhk7v\") pod \"payload-pre-processing-e2e-mt-678f625f-59c7bc5874-trjv6\" (UID: \"9fac4a7c-3089-4b7b-a111-fd10fa79f4df\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-678f625f-59c7bc5874-trjv6" Sep 10 15:59:38.608187 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:38.608013 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fac4a7c-3089-4b7b-a111-fd10fa79f4df-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-678f625f-59c7bc5874-trjv6\" (UID: \"9fac4a7c-3089-4b7b-a111-fd10fa79f4df\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-678f625f-59c7bc5874-trjv6" Sep 10 15:59:38.608187 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:38.608040 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c12953a-5e35-480c-8395-95cfd2b6c98c-plugins-config-volume\") pod \"payload-processing-e2e-mt-678f625f-dbcf47945-9c9kv\" (UID: \"2c12953a-5e35-480c-8395-95cfd2b6c98c\") " pod="openshift-ingress/payload-processing-e2e-mt-678f625f-dbcf47945-9c9kv" Sep 10 15:59:38.608187 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:38.608102 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jztbh\" (UniqueName: \"kubernetes.io/projected/2c12953a-5e35-480c-8395-95cfd2b6c98c-kube-api-access-jztbh\") pod \"payload-processing-e2e-mt-678f625f-dbcf47945-9c9kv\" (UID: \"2c12953a-5e35-480c-8395-95cfd2b6c98c\") " pod="openshift-ingress/payload-processing-e2e-mt-678f625f-dbcf47945-9c9kv" Sep 10 15:59:38.622490 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:38.622463 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:38.708804 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:38.708771 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhk7v\" (UniqueName: \"kubernetes.io/projected/9fac4a7c-3089-4b7b-a111-fd10fa79f4df-kube-api-access-mhk7v\") pod \"payload-pre-processing-e2e-mt-678f625f-59c7bc5874-trjv6\" (UID: \"9fac4a7c-3089-4b7b-a111-fd10fa79f4df\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-678f625f-59c7bc5874-trjv6" Sep 10 15:59:38.708997 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:38.708818 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fac4a7c-3089-4b7b-a111-fd10fa79f4df-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-678f625f-59c7bc5874-trjv6\" (UID: \"9fac4a7c-3089-4b7b-a111-fd10fa79f4df\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-678f625f-59c7bc5874-trjv6" Sep 10 15:59:38.708997 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:38.708837 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c12953a-5e35-480c-8395-95cfd2b6c98c-plugins-config-volume\") pod \"payload-processing-e2e-mt-678f625f-dbcf47945-9c9kv\" (UID: \"2c12953a-5e35-480c-8395-95cfd2b6c98c\") " pod="openshift-ingress/payload-processing-e2e-mt-678f625f-dbcf47945-9c9kv" Sep 10 15:59:38.708997 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:38.708861 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jztbh\" (UniqueName: \"kubernetes.io/projected/2c12953a-5e35-480c-8395-95cfd2b6c98c-kube-api-access-jztbh\") pod \"payload-processing-e2e-mt-678f625f-dbcf47945-9c9kv\" (UID: \"2c12953a-5e35-480c-8395-95cfd2b6c98c\") " pod="openshift-ingress/payload-processing-e2e-mt-678f625f-dbcf47945-9c9kv" Sep 10 15:59:38.709611 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:38.709585 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fac4a7c-3089-4b7b-a111-fd10fa79f4df-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-678f625f-59c7bc5874-trjv6\" (UID: \"9fac4a7c-3089-4b7b-a111-fd10fa79f4df\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-678f625f-59c7bc5874-trjv6" Sep 10 15:59:38.709702 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:38.709598 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c12953a-5e35-480c-8395-95cfd2b6c98c-plugins-config-volume\") pod \"payload-processing-e2e-mt-678f625f-dbcf47945-9c9kv\" (UID: \"2c12953a-5e35-480c-8395-95cfd2b6c98c\") " pod="openshift-ingress/payload-processing-e2e-mt-678f625f-dbcf47945-9c9kv" Sep 10 15:59:38.718809 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:38.718785 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jztbh\" (UniqueName: \"kubernetes.io/projected/2c12953a-5e35-480c-8395-95cfd2b6c98c-kube-api-access-jztbh\") pod \"payload-processing-e2e-mt-678f625f-dbcf47945-9c9kv\" (UID: \"2c12953a-5e35-480c-8395-95cfd2b6c98c\") " pod="openshift-ingress/payload-processing-e2e-mt-678f625f-dbcf47945-9c9kv" Sep 10 15:59:38.722542 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:38.722518 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhk7v\" (UniqueName: \"kubernetes.io/projected/9fac4a7c-3089-4b7b-a111-fd10fa79f4df-kube-api-access-mhk7v\") pod \"payload-pre-processing-e2e-mt-678f625f-59c7bc5874-trjv6\" (UID: \"9fac4a7c-3089-4b7b-a111-fd10fa79f4df\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-678f625f-59c7bc5874-trjv6" Sep 10 15:59:38.820871 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:38.820842 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-678f625f-dbcf47945-9c9kv" Sep 10 15:59:38.852215 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:38.852185 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-678f625f-59c7bc5874-trjv6" Sep 10 15:59:38.904575 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:38.904455 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:39.099714 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:39.099686 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-678f625f-dbcf47945-9c9kv"] Sep 10 15:59:39.101496 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:59:39.101471 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c12953a_5e35_480c_8395_95cfd2b6c98c.slice/crio-5bd79668036c7427014d95d3c9fc8ad6a9d6c2250d4c323856fd5ffa429d48cd WatchSource:0}: Error finding container 5bd79668036c7427014d95d3c9fc8ad6a9d6c2250d4c323856fd5ffa429d48cd: Status 404 returned error can't find the container with id 5bd79668036c7427014d95d3c9fc8ad6a9d6c2250d4c323856fd5ffa429d48cd Sep 10 15:59:39.145130 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:39.145105 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-678f625f-59c7bc5874-trjv6"] Sep 10 15:59:39.148669 ip-10-0-139-236 kubenswrapper[2576]: W0910 15:59:39.148647 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fac4a7c_3089_4b7b_a111_fd10fa79f4df.slice/crio-3cacbac7d3c7b35268dfa422c6d4f50d2136303cf1bb6ceaf99fbf3c391b3c02 WatchSource:0}: Error finding container 3cacbac7d3c7b35268dfa422c6d4f50d2136303cf1bb6ceaf99fbf3c391b3c02: Status 404 returned error can't find the container with id 3cacbac7d3c7b35268dfa422c6d4f50d2136303cf1bb6ceaf99fbf3c391b3c02 Sep 10 15:59:39.448180 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:39.448080 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-678f625f-dbcf47945-9c9kv" event={"ID":"2c12953a-5e35-480c-8395-95cfd2b6c98c","Type":"ContainerStarted","Data":"f41af2524eca0dbc52578c220a12c219e50d1fb87bfa69962fe85724457be137"} Sep 10 15:59:39.448180 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:39.448156 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-678f625f-dbcf47945-9c9kv" event={"ID":"2c12953a-5e35-480c-8395-95cfd2b6c98c","Type":"ContainerStarted","Data":"5bd79668036c7427014d95d3c9fc8ad6a9d6c2250d4c323856fd5ffa429d48cd"} Sep 10 15:59:39.448418 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:39.448281 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-678f625f-dbcf47945-9c9kv" Sep 10 15:59:39.449934 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:39.449911 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-678f625f-59c7bc5874-trjv6" event={"ID":"9fac4a7c-3089-4b7b-a111-fd10fa79f4df","Type":"ContainerStarted","Data":"3dc2dc56434fed0a3b27a515bda42d49ce8e343f23f3b2298fc476a15afba9c9"} Sep 10 15:59:39.450069 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:39.449956 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-678f625f-59c7bc5874-trjv6" event={"ID":"9fac4a7c-3089-4b7b-a111-fd10fa79f4df","Type":"ContainerStarted","Data":"3cacbac7d3c7b35268dfa422c6d4f50d2136303cf1bb6ceaf99fbf3c391b3c02"} Sep 10 15:59:39.450069 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:39.449993 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-678f625f-59c7bc5874-trjv6" Sep 10 15:59:39.471157 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:39.471119 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-678f625f-dbcf47945-9c9kv" podStartSLOduration=1.4711050669999999 podStartE2EDuration="1.471105067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:59:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:59:39.469807936 +0000 UTC m=+1056.823327122" watchObservedRunningTime="2026-09-10 15:59:39.471105067 +0000 UTC m=+1056.824624269" Sep 10 15:59:39.493910 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:39.489781 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-678f625f-59c7bc5874-trjv6" podStartSLOduration=1.489764704 podStartE2EDuration="1.489764704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:59:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:59:39.485793724 +0000 UTC m=+1056.839312903" watchObservedRunningTime="2026-09-10 15:59:39.489764704 +0000 UTC m=+1056.843283867" Sep 10 15:59:39.969134 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:39.969095 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:42.386823 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:42.386787 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 15:59:42.725698 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:42.725670 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-789be7-openshift-default-6669946986-bchwf" Sep 10 15:59:42.846225 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:42.846195 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d22ca135-fb1d-41c7-bba6-9109e30cfe61-istio-podinfo\") pod \"d22ca135-fb1d-41c7-bba6-9109e30cfe61\" (UID: \"d22ca135-fb1d-41c7-bba6-9109e30cfe61\") " Sep 10 15:59:42.846225 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:42.846224 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v7qnt\" (UniqueName: \"kubernetes.io/projected/d22ca135-fb1d-41c7-bba6-9109e30cfe61-kube-api-access-v7qnt\") pod \"d22ca135-fb1d-41c7-bba6-9109e30cfe61\" (UID: \"d22ca135-fb1d-41c7-bba6-9109e30cfe61\") " Sep 10 15:59:42.846438 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:42.846272 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d22ca135-fb1d-41c7-bba6-9109e30cfe61-istiod-ca-cert\") pod \"d22ca135-fb1d-41c7-bba6-9109e30cfe61\" (UID: \"d22ca135-fb1d-41c7-bba6-9109e30cfe61\") " Sep 10 15:59:42.846438 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:42.846290 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d22ca135-fb1d-41c7-bba6-9109e30cfe61-workload-certs\") pod \"d22ca135-fb1d-41c7-bba6-9109e30cfe61\" (UID: \"d22ca135-fb1d-41c7-bba6-9109e30cfe61\") " Sep 10 15:59:42.846438 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:42.846307 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d22ca135-fb1d-41c7-bba6-9109e30cfe61-istio-data\") pod \"d22ca135-fb1d-41c7-bba6-9109e30cfe61\" (UID: \"d22ca135-fb1d-41c7-bba6-9109e30cfe61\") " Sep 10 15:59:42.846438 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:42.846377 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d22ca135-fb1d-41c7-bba6-9109e30cfe61-credential-socket\") pod \"d22ca135-fb1d-41c7-bba6-9109e30cfe61\" (UID: \"d22ca135-fb1d-41c7-bba6-9109e30cfe61\") " Sep 10 15:59:42.846438 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:42.846407 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d22ca135-fb1d-41c7-bba6-9109e30cfe61-istio-token\") pod \"d22ca135-fb1d-41c7-bba6-9109e30cfe61\" (UID: \"d22ca135-fb1d-41c7-bba6-9109e30cfe61\") " Sep 10 15:59:42.846716 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:42.846454 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d22ca135-fb1d-41c7-bba6-9109e30cfe61-istio-envoy\") pod \"d22ca135-fb1d-41c7-bba6-9109e30cfe61\" (UID: \"d22ca135-fb1d-41c7-bba6-9109e30cfe61\") " Sep 10 15:59:42.846716 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:42.846483 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d22ca135-fb1d-41c7-bba6-9109e30cfe61-workload-socket\") pod \"d22ca135-fb1d-41c7-bba6-9109e30cfe61\" (UID: \"d22ca135-fb1d-41c7-bba6-9109e30cfe61\") " Sep 10 15:59:42.846716 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:42.846616 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d22ca135-fb1d-41c7-bba6-9109e30cfe61-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d22ca135-fb1d-41c7-bba6-9109e30cfe61" (UID: "d22ca135-fb1d-41c7-bba6-9109e30cfe61"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:59:42.846716 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:42.846682 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d22ca135-fb1d-41c7-bba6-9109e30cfe61-istio-data" (OuterVolumeSpecName: "istio-data") pod "d22ca135-fb1d-41c7-bba6-9109e30cfe61" (UID: "d22ca135-fb1d-41c7-bba6-9109e30cfe61"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:59:42.846716 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:42.846707 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d22ca135-fb1d-41c7-bba6-9109e30cfe61-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d22ca135-fb1d-41c7-bba6-9109e30cfe61" (UID: "d22ca135-fb1d-41c7-bba6-9109e30cfe61"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:59:42.847021 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:42.846867 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d22ca135-fb1d-41c7-bba6-9109e30cfe61-istiod-ca-cert\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:59:42.847021 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:42.846885 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d22ca135-fb1d-41c7-bba6-9109e30cfe61-workload-certs\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:59:42.847021 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:42.846897 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d22ca135-fb1d-41c7-bba6-9109e30cfe61-istio-data\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:59:42.847021 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:42.846940 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d22ca135-fb1d-41c7-bba6-9109e30cfe61-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d22ca135-fb1d-41c7-bba6-9109e30cfe61" (UID: "d22ca135-fb1d-41c7-bba6-9109e30cfe61"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:59:42.847021 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:42.846954 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d22ca135-fb1d-41c7-bba6-9109e30cfe61-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d22ca135-fb1d-41c7-bba6-9109e30cfe61" (UID: "d22ca135-fb1d-41c7-bba6-9109e30cfe61"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:59:42.848682 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:42.848629 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d22ca135-fb1d-41c7-bba6-9109e30cfe61-istio-token" (OuterVolumeSpecName: "istio-token") pod "d22ca135-fb1d-41c7-bba6-9109e30cfe61" (UID: "d22ca135-fb1d-41c7-bba6-9109e30cfe61"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:59:42.848832 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:42.848685 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d22ca135-fb1d-41c7-bba6-9109e30cfe61-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d22ca135-fb1d-41c7-bba6-9109e30cfe61" (UID: "d22ca135-fb1d-41c7-bba6-9109e30cfe61"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 15:59:42.848832 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:42.848689 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d22ca135-fb1d-41c7-bba6-9109e30cfe61-kube-api-access-v7qnt" (OuterVolumeSpecName: "kube-api-access-v7qnt") pod "d22ca135-fb1d-41c7-bba6-9109e30cfe61" (UID: "d22ca135-fb1d-41c7-bba6-9109e30cfe61"). InnerVolumeSpecName "kube-api-access-v7qnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:59:42.848961 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:42.848925 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d22ca135-fb1d-41c7-bba6-9109e30cfe61-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d22ca135-fb1d-41c7-bba6-9109e30cfe61" (UID: "d22ca135-fb1d-41c7-bba6-9109e30cfe61"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:59:42.947614 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:42.947584 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d22ca135-fb1d-41c7-bba6-9109e30cfe61-istio-envoy\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:59:42.947614 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:42.947611 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d22ca135-fb1d-41c7-bba6-9109e30cfe61-workload-socket\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:59:42.947614 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:42.947620 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d22ca135-fb1d-41c7-bba6-9109e30cfe61-istio-podinfo\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:59:42.947861 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:42.947629 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7qnt\" (UniqueName: \"kubernetes.io/projected/d22ca135-fb1d-41c7-bba6-9109e30cfe61-kube-api-access-v7qnt\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:59:42.947861 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:42.947638 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d22ca135-fb1d-41c7-bba6-9109e30cfe61-credential-socket\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:59:42.947861 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:42.947646 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d22ca135-fb1d-41c7-bba6-9109e30cfe61-istio-token\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:59:43.471373 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:43.471287 2576 generic.go:356] "Generic (PLEG): container finished" podID="d22ca135-fb1d-41c7-bba6-9109e30cfe61" containerID="d14c027af6ffee6e7bd5c45dca0e656b1bd8a899b84462474dbcf4e580f38213" exitCode=0 Sep 10 15:59:43.471373 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:43.471338 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-789be7-openshift-default-6669946986-bchwf" Sep 10 15:59:43.471373 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:43.471334 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-789be7-openshift-default-6669946986-bchwf" event={"ID":"d22ca135-fb1d-41c7-bba6-9109e30cfe61","Type":"ContainerDied","Data":"d14c027af6ffee6e7bd5c45dca0e656b1bd8a899b84462474dbcf4e580f38213"} Sep 10 15:59:43.471953 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:43.471402 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-789be7-openshift-default-6669946986-bchwf" event={"ID":"d22ca135-fb1d-41c7-bba6-9109e30cfe61","Type":"ContainerDied","Data":"f52f4d03491577b4862ce98dcc9131b5d6fd917ad66530bc78285fc2ea5479bf"} Sep 10 15:59:43.471953 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:43.471433 2576 scope.go:117] "RemoveContainer" containerID="d14c027af6ffee6e7bd5c45dca0e656b1bd8a899b84462474dbcf4e580f38213" Sep 10 15:59:43.480465 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:43.480451 2576 scope.go:117] "RemoveContainer" containerID="eb2701de50319a1d3a406f55e998e80620caec0dd4623b315a2e37cf4731fb4e" Sep 10 15:59:43.488453 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:43.488261 2576 scope.go:117] "RemoveContainer" containerID="d14c027af6ffee6e7bd5c45dca0e656b1bd8a899b84462474dbcf4e580f38213" Sep 10 15:59:43.488615 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:59:43.488583 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d14c027af6ffee6e7bd5c45dca0e656b1bd8a899b84462474dbcf4e580f38213\": container with ID starting with d14c027af6ffee6e7bd5c45dca0e656b1bd8a899b84462474dbcf4e580f38213 not found: ID does not exist" containerID="d14c027af6ffee6e7bd5c45dca0e656b1bd8a899b84462474dbcf4e580f38213" Sep 10 15:59:43.488770 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:43.488614 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d14c027af6ffee6e7bd5c45dca0e656b1bd8a899b84462474dbcf4e580f38213"} err="failed to get container status \"d14c027af6ffee6e7bd5c45dca0e656b1bd8a899b84462474dbcf4e580f38213\": rpc error: code = NotFound desc = could not find container \"d14c027af6ffee6e7bd5c45dca0e656b1bd8a899b84462474dbcf4e580f38213\": container with ID starting with d14c027af6ffee6e7bd5c45dca0e656b1bd8a899b84462474dbcf4e580f38213 not found: ID does not exist" Sep 10 15:59:43.488770 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:43.488635 2576 scope.go:117] "RemoveContainer" containerID="eb2701de50319a1d3a406f55e998e80620caec0dd4623b315a2e37cf4731fb4e" Sep 10 15:59:43.488962 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:59:43.488931 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb2701de50319a1d3a406f55e998e80620caec0dd4623b315a2e37cf4731fb4e\": container with ID starting with eb2701de50319a1d3a406f55e998e80620caec0dd4623b315a2e37cf4731fb4e not found: ID does not exist" containerID="eb2701de50319a1d3a406f55e998e80620caec0dd4623b315a2e37cf4731fb4e" Sep 10 15:59:43.489038 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:43.488958 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb2701de50319a1d3a406f55e998e80620caec0dd4623b315a2e37cf4731fb4e"} err="failed to get container status \"eb2701de50319a1d3a406f55e998e80620caec0dd4623b315a2e37cf4731fb4e\": rpc error: code = NotFound desc = could not find container \"eb2701de50319a1d3a406f55e998e80620caec0dd4623b315a2e37cf4731fb4e\": container with ID starting with eb2701de50319a1d3a406f55e998e80620caec0dd4623b315a2e37cf4731fb4e not found: ID does not exist" Sep 10 15:59:43.490937 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:43.490916 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-789be7-openshift-default-6669946986-bchwf"] Sep 10 15:59:43.498434 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:43.498414 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-789be7-openshift-default-6669946986-bchwf"] Sep 10 15:59:45.169264 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:45.169232 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d22ca135-fb1d-41c7-bba6-9109e30cfe61" path="/var/lib/kubelet/pods/d22ca135-fb1d-41c7-bba6-9109e30cfe61/volumes" Sep 10 15:59:48.026899 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:48.026862 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:48.890137 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:48.890102 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5d6f34-7ffc476787-xcx97"] Sep 10 15:59:48.890408 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:48.890376 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-5d6f34-7ffc476787-xcx97" podUID="5462ffa6-e222-4771-add5-6afcf521ce8f" containerName="payload-processing" containerID="cri-o://1da1fc448a1eae5f945d82d98f34e16ac35e96faf5935d7a44e39c16bede035b" gracePeriod=30 Sep 10 15:59:49.226193 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:49.226163 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5d6f34-7ffc476787-xcx97" Sep 10 15:59:49.329649 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:49.329621 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qg4lf\" (UniqueName: \"kubernetes.io/projected/5462ffa6-e222-4771-add5-6afcf521ce8f-kube-api-access-qg4lf\") pod \"5462ffa6-e222-4771-add5-6afcf521ce8f\" (UID: \"5462ffa6-e222-4771-add5-6afcf521ce8f\") " Sep 10 15:59:49.329848 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:49.329713 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5462ffa6-e222-4771-add5-6afcf521ce8f-plugins-config-volume\") pod \"5462ffa6-e222-4771-add5-6afcf521ce8f\" (UID: \"5462ffa6-e222-4771-add5-6afcf521ce8f\") " Sep 10 15:59:49.330039 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:49.330018 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5462ffa6-e222-4771-add5-6afcf521ce8f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5462ffa6-e222-4771-add5-6afcf521ce8f" (UID: "5462ffa6-e222-4771-add5-6afcf521ce8f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:59:49.331943 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:49.331922 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5462ffa6-e222-4771-add5-6afcf521ce8f-kube-api-access-qg4lf" (OuterVolumeSpecName: "kube-api-access-qg4lf") pod "5462ffa6-e222-4771-add5-6afcf521ce8f" (UID: "5462ffa6-e222-4771-add5-6afcf521ce8f"). InnerVolumeSpecName "kube-api-access-qg4lf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:59:49.392624 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:49.392597 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:49.430535 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:49.430471 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qg4lf\" (UniqueName: \"kubernetes.io/projected/5462ffa6-e222-4771-add5-6afcf521ce8f-kube-api-access-qg4lf\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:59:49.430535 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:49.430499 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5462ffa6-e222-4771-add5-6afcf521ce8f-plugins-config-volume\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 15:59:49.510510 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:49.510476 2576 generic.go:356] "Generic (PLEG): container finished" podID="5462ffa6-e222-4771-add5-6afcf521ce8f" containerID="1da1fc448a1eae5f945d82d98f34e16ac35e96faf5935d7a44e39c16bede035b" exitCode=0 Sep 10 15:59:49.510766 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:49.510726 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5d6f34-7ffc476787-xcx97" Sep 10 15:59:49.510968 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:49.510650 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5d6f34-7ffc476787-xcx97" event={"ID":"5462ffa6-e222-4771-add5-6afcf521ce8f","Type":"ContainerDied","Data":"1da1fc448a1eae5f945d82d98f34e16ac35e96faf5935d7a44e39c16bede035b"} Sep 10 15:59:49.511108 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:49.511091 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5d6f34-7ffc476787-xcx97" event={"ID":"5462ffa6-e222-4771-add5-6afcf521ce8f","Type":"ContainerDied","Data":"abd5310d6d5a623f3cb82ce196eb09940cf31c9f97fc857950f4cc8a63cbf528"} Sep 10 15:59:49.511242 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:49.511227 2576 scope.go:117] "RemoveContainer" containerID="1da1fc448a1eae5f945d82d98f34e16ac35e96faf5935d7a44e39c16bede035b" Sep 10 15:59:49.523723 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:49.523706 2576 scope.go:117] "RemoveContainer" containerID="1da1fc448a1eae5f945d82d98f34e16ac35e96faf5935d7a44e39c16bede035b" Sep 10 15:59:49.524328 ip-10-0-139-236 kubenswrapper[2576]: E0910 15:59:49.524305 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1da1fc448a1eae5f945d82d98f34e16ac35e96faf5935d7a44e39c16bede035b\": container with ID starting with 1da1fc448a1eae5f945d82d98f34e16ac35e96faf5935d7a44e39c16bede035b not found: ID does not exist" containerID="1da1fc448a1eae5f945d82d98f34e16ac35e96faf5935d7a44e39c16bede035b" Sep 10 15:59:49.524464 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:49.524442 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1da1fc448a1eae5f945d82d98f34e16ac35e96faf5935d7a44e39c16bede035b"} err="failed to get container status \"1da1fc448a1eae5f945d82d98f34e16ac35e96faf5935d7a44e39c16bede035b\": rpc error: code = NotFound desc = could not find container \"1da1fc448a1eae5f945d82d98f34e16ac35e96faf5935d7a44e39c16bede035b\": container with ID starting with 1da1fc448a1eae5f945d82d98f34e16ac35e96faf5935d7a44e39c16bede035b not found: ID does not exist" Sep 10 15:59:49.548215 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:49.548190 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5d6f34-7ffc476787-xcx97"] Sep 10 15:59:49.553946 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:49.553926 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5d6f34-7ffc476787-xcx97"] Sep 10 15:59:49.616533 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:49.616501 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:50.457458 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:50.457386 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-678f625f-59c7bc5874-trjv6" Sep 10 15:59:50.457458 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:50.457452 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-678f625f-dbcf47945-9c9kv" Sep 10 15:59:51.169746 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:51.169700 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5462ffa6-e222-4771-add5-6afcf521ce8f" path="/var/lib/kubelet/pods/5462ffa6-e222-4771-add5-6afcf521ce8f/volumes" Sep 10 15:59:52.471904 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:52.471862 2576 patch_prober.go:28] interesting pod/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 15:59:52.472256 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:52.471935 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj" podUID="1d500885-c94a-4f5e-b9da-61e310e6ad7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 15:59:52.799226 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:52.799197 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 15:59:53.535097 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:53.535018 2576 generic.go:356] "Generic (PLEG): container finished" podID="1d500885-c94a-4f5e-b9da-61e310e6ad7c" containerID="3513b9c880370e635603a4f9290c86b5a151b96a9b05f94148a8b59cfe99707d" exitCode=0 Sep 10 15:59:53.535510 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:53.535087 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj" event={"ID":"1d500885-c94a-4f5e-b9da-61e310e6ad7c","Type":"ContainerDied","Data":"3513b9c880370e635603a4f9290c86b5a151b96a9b05f94148a8b59cfe99707d"} Sep 10 15:59:53.535510 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:53.535455 2576 scope.go:117] "RemoveContainer" containerID="3513b9c880370e635603a4f9290c86b5a151b96a9b05f94148a8b59cfe99707d" Sep 10 15:59:53.536475 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:53.536451 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 15:59:53.536528 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:53.536517 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 15:59:53.536564 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:53.536553 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 15:59:54.542506 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:54.542427 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj" event={"ID":"1d500885-c94a-4f5e-b9da-61e310e6ad7c","Type":"ContainerStarted","Data":"23731ea95d12bc3094b647e16659b0f39e2c11305526c58f80daf785180b2e34"} Sep 10 15:59:54.559989 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:54.559957 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj" Sep 10 15:59:54.560310 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:54.560284 2576 patch_prober.go:28] interesting pod/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Sep 10 15:59:54.560360 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:54.560342 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj" podUID="1d500885-c94a-4f5e-b9da-61e310e6ad7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Sep 10 15:59:55.559578 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:55.559540 2576 patch_prober.go:28] interesting pod/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Sep 10 15:59:55.559984 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:55.559594 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj" podUID="1d500885-c94a-4f5e-b9da-61e310e6ad7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Sep 10 15:59:55.714651 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:55.714547 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:55.816626 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:55.816506 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:56.014245 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:56.014199 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:56.560348 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:56.560307 2576 patch_prober.go:28] interesting pod/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Sep 10 15:59:56.560752 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:56.560384 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj" podUID="1d500885-c94a-4f5e-b9da-61e310e6ad7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Sep 10 15:59:56.606603 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:56.606511 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:56.937033 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:56.936935 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:57.082232 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:57.082194 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:57.560406 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:57.560375 2576 patch_prober.go:28] interesting pod/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Sep 10 15:59:57.560816 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:57.560432 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj" podUID="1d500885-c94a-4f5e-b9da-61e310e6ad7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Sep 10 15:59:58.560071 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:58.560037 2576 patch_prober.go:28] interesting pod/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Sep 10 15:59:58.560249 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:58.560101 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj" podUID="1d500885-c94a-4f5e-b9da-61e310e6ad7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Sep 10 15:59:59.001049 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:59.000950 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:59.544305 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:59.544267 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:59.574944 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:59.574898 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj" Sep 10 15:59:59.575352 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:59.575334 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj" Sep 10 15:59:59.577169 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:59.577146 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj" Sep 10 15:59:59.658451 ip-10-0-139-236 kubenswrapper[2576]: I0910 15:59:59.657800 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:00.150459 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:00.150427 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-678f625f-29817600-lhvms"] Sep 10 16:00:00.150904 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:00.150890 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d22ca135-fb1d-41c7-bba6-9109e30cfe61" containerName="istio-proxy" Sep 10 16:00:00.150958 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:00.150906 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d22ca135-fb1d-41c7-bba6-9109e30cfe61" containerName="istio-proxy" Sep 10 16:00:00.150958 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:00.150915 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5462ffa6-e222-4771-add5-6afcf521ce8f" containerName="payload-processing" Sep 10 16:00:00.150958 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:00.150921 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5462ffa6-e222-4771-add5-6afcf521ce8f" containerName="payload-processing" Sep 10 16:00:00.150958 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:00.150928 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d22ca135-fb1d-41c7-bba6-9109e30cfe61" containerName="istio-proxy" Sep 10 16:00:00.150958 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:00.150934 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d22ca135-fb1d-41c7-bba6-9109e30cfe61" containerName="istio-proxy" Sep 10 16:00:00.151154 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:00.150997 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d22ca135-fb1d-41c7-bba6-9109e30cfe61" containerName="istio-proxy" Sep 10 16:00:00.151154 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:00.151006 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d22ca135-fb1d-41c7-bba6-9109e30cfe61" containerName="istio-proxy" Sep 10 16:00:00.151154 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:00.151012 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5462ffa6-e222-4771-add5-6afcf521ce8f" containerName="payload-processing" Sep 10 16:00:00.153948 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:00.153931 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-678f625f-29817600-lhvms" Sep 10 16:00:00.157520 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:00.157496 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwwht\"" Sep 10 16:00:00.163401 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:00.163381 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-678f625f-29817600-lhvms" Sep 10 16:00:00.191257 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:00.191223 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-678f625f-29817600-lhvms"] Sep 10 16:00:00.443034 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:00.443009 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-678f625f-29817600-lhvms"] Sep 10 16:00:00.445746 ip-10-0-139-236 kubenswrapper[2576]: W0910 16:00:00.445710 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc816a64d_aab1_44a1_8209_9548164db02f.slice/crio-28b9ffcc1373738433437aba4ef42a9a8b146675490fbc8edc5625f612d3e940 WatchSource:0}: Error finding container 28b9ffcc1373738433437aba4ef42a9a8b146675490fbc8edc5625f612d3e940: Status 404 returned error can't find the container with id 28b9ffcc1373738433437aba4ef42a9a8b146675490fbc8edc5625f612d3e940 Sep 10 16:00:00.580538 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:00.580496 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-678f625f-29817600-lhvms" event={"ID":"c816a64d-aab1-44a1-8209-9548164db02f","Type":"ContainerStarted","Data":"28b9ffcc1373738433437aba4ef42a9a8b146675490fbc8edc5625f612d3e940"} Sep 10 16:00:01.273411 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:01.273377 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:01.520442 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:01.520412 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:01.587859 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:01.587822 2576 generic.go:356] "Generic (PLEG): container finished" podID="c816a64d-aab1-44a1-8209-9548164db02f" containerID="610db5d44c9a522e550c5130d92e152f4c7ba6ef1fb13ba90917e208cc83c474" exitCode=0 Sep 10 16:00:01.588051 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:01.587864 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-678f625f-29817600-lhvms" event={"ID":"c816a64d-aab1-44a1-8209-9548164db02f","Type":"ContainerDied","Data":"610db5d44c9a522e550c5130d92e152f4c7ba6ef1fb13ba90917e208cc83c474"} Sep 10 16:00:01.626328 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:01.626293 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:01.945917 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:01.945825 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:02.040739 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:02.040710 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-50ed73-657b6f9cdb-4xh7x"] Sep 10 16:00:02.044325 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:02.044309 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-50ed73-657b6f9cdb-4xh7x" Sep 10 16:00:02.046524 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:02.046501 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-50ed73\"" Sep 10 16:00:02.066014 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:02.065991 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-50ed73-657b6f9cdb-4xh7x"] Sep 10 16:00:02.074459 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:02.074436 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-50ed73-7c8bb55487-kpbvm"] Sep 10 16:00:02.077898 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:02.077877 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-50ed73-7c8bb55487-kpbvm" Sep 10 16:00:02.080262 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:02.080244 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-50ed73-dockercfg-7scw6\"" Sep 10 16:00:02.110617 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:02.110583 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-50ed73-7c8bb55487-kpbvm"] Sep 10 16:00:02.160946 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:02.160922 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecfb8c74-1806-43d3-8d8d-573617251bb9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-50ed73-7c8bb55487-kpbvm\" (UID: \"ecfb8c74-1806-43d3-8d8d-573617251bb9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-50ed73-7c8bb55487-kpbvm" Sep 10 16:00:02.161060 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:02.160957 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz4cw\" (UniqueName: \"kubernetes.io/projected/c6c987d0-fdd8-4180-9542-1020e83a704d-kube-api-access-fz4cw\") pod \"payload-pre-processing-e2e-shared-a-w3-50ed73-657b6f9cdb-4xh7x\" (UID: \"c6c987d0-fdd8-4180-9542-1020e83a704d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-50ed73-657b6f9cdb-4xh7x" Sep 10 16:00:02.161060 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:02.161044 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6c987d0-fdd8-4180-9542-1020e83a704d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-50ed73-657b6f9cdb-4xh7x\" (UID: \"c6c987d0-fdd8-4180-9542-1020e83a704d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-50ed73-657b6f9cdb-4xh7x" Sep 10 16:00:02.161136 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:02.161109 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8c2v\" (UniqueName: \"kubernetes.io/projected/ecfb8c74-1806-43d3-8d8d-573617251bb9-kube-api-access-j8c2v\") pod \"payload-processing-e2e-shared-a-w3-50ed73-7c8bb55487-kpbvm\" (UID: \"ecfb8c74-1806-43d3-8d8d-573617251bb9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-50ed73-7c8bb55487-kpbvm" Sep 10 16:00:02.261971 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:02.261887 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecfb8c74-1806-43d3-8d8d-573617251bb9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-50ed73-7c8bb55487-kpbvm\" (UID: \"ecfb8c74-1806-43d3-8d8d-573617251bb9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-50ed73-7c8bb55487-kpbvm" Sep 10 16:00:02.261971 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:02.261929 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz4cw\" (UniqueName: \"kubernetes.io/projected/c6c987d0-fdd8-4180-9542-1020e83a704d-kube-api-access-fz4cw\") pod \"payload-pre-processing-e2e-shared-a-w3-50ed73-657b6f9cdb-4xh7x\" (UID: \"c6c987d0-fdd8-4180-9542-1020e83a704d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-50ed73-657b6f9cdb-4xh7x" Sep 10 16:00:02.262196 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:02.262029 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6c987d0-fdd8-4180-9542-1020e83a704d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-50ed73-657b6f9cdb-4xh7x\" (UID: \"c6c987d0-fdd8-4180-9542-1020e83a704d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-50ed73-657b6f9cdb-4xh7x" Sep 10 16:00:02.262196 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:02.262089 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8c2v\" (UniqueName: \"kubernetes.io/projected/ecfb8c74-1806-43d3-8d8d-573617251bb9-kube-api-access-j8c2v\") pod \"payload-processing-e2e-shared-a-w3-50ed73-7c8bb55487-kpbvm\" (UID: \"ecfb8c74-1806-43d3-8d8d-573617251bb9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-50ed73-7c8bb55487-kpbvm" Sep 10 16:00:02.262763 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:02.262719 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecfb8c74-1806-43d3-8d8d-573617251bb9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-50ed73-7c8bb55487-kpbvm\" (UID: \"ecfb8c74-1806-43d3-8d8d-573617251bb9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-50ed73-7c8bb55487-kpbvm" Sep 10 16:00:02.262984 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:02.262960 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6c987d0-fdd8-4180-9542-1020e83a704d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-50ed73-657b6f9cdb-4xh7x\" (UID: \"c6c987d0-fdd8-4180-9542-1020e83a704d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-50ed73-657b6f9cdb-4xh7x" Sep 10 16:00:02.273865 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:02.273837 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz4cw\" (UniqueName: \"kubernetes.io/projected/c6c987d0-fdd8-4180-9542-1020e83a704d-kube-api-access-fz4cw\") pod \"payload-pre-processing-e2e-shared-a-w3-50ed73-657b6f9cdb-4xh7x\" (UID: \"c6c987d0-fdd8-4180-9542-1020e83a704d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-50ed73-657b6f9cdb-4xh7x" Sep 10 16:00:02.273865 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:02.273852 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8c2v\" (UniqueName: \"kubernetes.io/projected/ecfb8c74-1806-43d3-8d8d-573617251bb9-kube-api-access-j8c2v\") pod \"payload-processing-e2e-shared-a-w3-50ed73-7c8bb55487-kpbvm\" (UID: \"ecfb8c74-1806-43d3-8d8d-573617251bb9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-50ed73-7c8bb55487-kpbvm" Sep 10 16:00:02.282500 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:02.282465 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:02.354536 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:02.354503 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-50ed73-657b6f9cdb-4xh7x" Sep 10 16:00:02.388215 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:02.388179 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-50ed73-7c8bb55487-kpbvm" Sep 10 16:00:02.459447 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:02.459142 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:02.631406 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:02.631382 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-50ed73-657b6f9cdb-4xh7x"] Sep 10 16:00:02.633930 ip-10-0-139-236 kubenswrapper[2576]: W0910 16:00:02.633899 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6c987d0_fdd8_4180_9542_1020e83a704d.slice/crio-126b1e502aa42dd24159620d83eb552aa2202c22e12bdc19d231b58fa727b47c WatchSource:0}: Error finding container 126b1e502aa42dd24159620d83eb552aa2202c22e12bdc19d231b58fa727b47c: Status 404 returned error can't find the container with id 126b1e502aa42dd24159620d83eb552aa2202c22e12bdc19d231b58fa727b47c Sep 10 16:00:02.687776 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:02.687717 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-50ed73-7c8bb55487-kpbvm"] Sep 10 16:00:02.690018 ip-10-0-139-236 kubenswrapper[2576]: W0910 16:00:02.689986 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecfb8c74_1806_43d3_8d8d_573617251bb9.slice/crio-67a77ef6fb0cf38c0dfb71b3acd89f81da2656f12c87808dcfe4a783fa44f621 WatchSource:0}: Error finding container 67a77ef6fb0cf38c0dfb71b3acd89f81da2656f12c87808dcfe4a783fa44f621: Status 404 returned error can't find the container with id 67a77ef6fb0cf38c0dfb71b3acd89f81da2656f12c87808dcfe4a783fa44f621 Sep 10 16:00:02.709829 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:02.709802 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:02.728725 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:02.728704 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-678f625f-29817600-lhvms" Sep 10 16:00:03.249647 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:03.249603 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:03.601056 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:03.601019 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-50ed73-7c8bb55487-kpbvm" event={"ID":"ecfb8c74-1806-43d3-8d8d-573617251bb9","Type":"ContainerStarted","Data":"bebe906e4d8dcc38a67a9310429fce2f20de4d17b5e98f9434fbdad6a7a9f573"} Sep 10 16:00:03.601583 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:03.601084 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-50ed73-7c8bb55487-kpbvm" Sep 10 16:00:03.601583 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:03.601107 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-50ed73-7c8bb55487-kpbvm" event={"ID":"ecfb8c74-1806-43d3-8d8d-573617251bb9","Type":"ContainerStarted","Data":"67a77ef6fb0cf38c0dfb71b3acd89f81da2656f12c87808dcfe4a783fa44f621"} Sep 10 16:00:03.602431 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:03.602405 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-678f625f-29817600-lhvms" Sep 10 16:00:03.602553 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:03.602452 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-678f625f-29817600-lhvms" event={"ID":"c816a64d-aab1-44a1-8209-9548164db02f","Type":"ContainerDied","Data":"28b9ffcc1373738433437aba4ef42a9a8b146675490fbc8edc5625f612d3e940"} Sep 10 16:00:03.602553 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:03.602496 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28b9ffcc1373738433437aba4ef42a9a8b146675490fbc8edc5625f612d3e940" Sep 10 16:00:03.604308 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:03.604285 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-50ed73-657b6f9cdb-4xh7x" event={"ID":"c6c987d0-fdd8-4180-9542-1020e83a704d","Type":"ContainerStarted","Data":"de2dfe232787b7b0788756e1a9e5aa9b0fd67ed6509bfee7057fb71a48ec250b"} Sep 10 16:00:03.604412 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:03.604326 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-50ed73-657b6f9cdb-4xh7x" event={"ID":"c6c987d0-fdd8-4180-9542-1020e83a704d","Type":"ContainerStarted","Data":"126b1e502aa42dd24159620d83eb552aa2202c22e12bdc19d231b58fa727b47c"} Sep 10 16:00:03.604541 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:03.604523 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-50ed73-657b6f9cdb-4xh7x" Sep 10 16:00:03.633393 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:03.633350 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-50ed73-7c8bb55487-kpbvm" podStartSLOduration=1.6333353050000001 podStartE2EDuration="1.633335305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:00:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:00:03.631405024 +0000 UTC m=+1080.984924186" watchObservedRunningTime="2026-09-10 16:00:03.633335305 +0000 UTC m=+1080.986854477" Sep 10 16:00:03.654566 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:03.654522 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-50ed73-657b6f9cdb-4xh7x" podStartSLOduration=1.654505581 podStartE2EDuration="1.654505581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:00:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:00:03.652235672 +0000 UTC m=+1081.005754856" watchObservedRunningTime="2026-09-10 16:00:03.654505581 +0000 UTC m=+1081.008024743" Sep 10 16:00:05.159515 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:05.159474 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:05.258532 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:05.258502 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:05.314114 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:05.314071 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 16:00:09.549443 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:09.549402 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:09.944009 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:09.943924 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:10.058928 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:10.058890 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:14.612646 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:14.612614 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-50ed73-7c8bb55487-kpbvm" Sep 10 16:00:14.613181 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:14.612797 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-50ed73-657b6f9cdb-4xh7x" Sep 10 16:00:20.548527 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:20.548495 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-678f625f-29817600-lhvms"] Sep 10 16:00:20.550862 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:20.550838 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-678f625f-29817600-lhvms"] Sep 10 16:00:20.552033 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:20.552003 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:20.590711 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:20.590679 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-678f625f-dbcf47945-9c9kv"] Sep 10 16:00:20.590992 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:20.590961 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-678f625f-dbcf47945-9c9kv" podUID="2c12953a-5e35-480c-8395-95cfd2b6c98c" containerName="payload-processing" containerID="cri-o://f41af2524eca0dbc52578c220a12c219e50d1fb87bfa69962fe85724457be137" gracePeriod=30 Sep 10 16:00:20.604130 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:20.604097 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-678f625f-59c7bc5874-trjv6"] Sep 10 16:00:20.604361 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:20.604334 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-678f625f-59c7bc5874-trjv6" podUID="9fac4a7c-3089-4b7b-a111-fd10fa79f4df" containerName="payload-pre-processing" containerID="cri-o://3dc2dc56434fed0a3b27a515bda42d49ce8e343f23f3b2298fc476a15afba9c9" gracePeriod=30 Sep 10 16:00:20.649850 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:20.649808 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:20.851448 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:20.851425 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-678f625f-dbcf47945-9c9kv" Sep 10 16:00:20.886409 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:20.886371 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-678f625f-59c7bc5874-trjv6" Sep 10 16:00:20.940451 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:20.940413 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c12953a-5e35-480c-8395-95cfd2b6c98c-plugins-config-volume\") pod \"2c12953a-5e35-480c-8395-95cfd2b6c98c\" (UID: \"2c12953a-5e35-480c-8395-95cfd2b6c98c\") " Sep 10 16:00:20.940637 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:20.940501 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jztbh\" (UniqueName: \"kubernetes.io/projected/2c12953a-5e35-480c-8395-95cfd2b6c98c-kube-api-access-jztbh\") pod \"2c12953a-5e35-480c-8395-95cfd2b6c98c\" (UID: \"2c12953a-5e35-480c-8395-95cfd2b6c98c\") " Sep 10 16:00:20.940925 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:20.940887 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c12953a-5e35-480c-8395-95cfd2b6c98c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2c12953a-5e35-480c-8395-95cfd2b6c98c" (UID: "2c12953a-5e35-480c-8395-95cfd2b6c98c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:00:20.942721 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:20.942690 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c12953a-5e35-480c-8395-95cfd2b6c98c-kube-api-access-jztbh" (OuterVolumeSpecName: "kube-api-access-jztbh") pod "2c12953a-5e35-480c-8395-95cfd2b6c98c" (UID: "2c12953a-5e35-480c-8395-95cfd2b6c98c"). InnerVolumeSpecName "kube-api-access-jztbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:00:21.041396 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:21.041368 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fac4a7c-3089-4b7b-a111-fd10fa79f4df-plugins-config-volume\") pod \"9fac4a7c-3089-4b7b-a111-fd10fa79f4df\" (UID: \"9fac4a7c-3089-4b7b-a111-fd10fa79f4df\") " Sep 10 16:00:21.041592 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:21.041475 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mhk7v\" (UniqueName: \"kubernetes.io/projected/9fac4a7c-3089-4b7b-a111-fd10fa79f4df-kube-api-access-mhk7v\") pod \"9fac4a7c-3089-4b7b-a111-fd10fa79f4df\" (UID: \"9fac4a7c-3089-4b7b-a111-fd10fa79f4df\") " Sep 10 16:00:21.041772 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:21.041728 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jztbh\" (UniqueName: \"kubernetes.io/projected/2c12953a-5e35-480c-8395-95cfd2b6c98c-kube-api-access-jztbh\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:00:21.041846 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:21.041775 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c12953a-5e35-480c-8395-95cfd2b6c98c-plugins-config-volume\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:00:21.041903 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:21.041836 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fac4a7c-3089-4b7b-a111-fd10fa79f4df-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9fac4a7c-3089-4b7b-a111-fd10fa79f4df" (UID: "9fac4a7c-3089-4b7b-a111-fd10fa79f4df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:00:21.043568 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:21.043547 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fac4a7c-3089-4b7b-a111-fd10fa79f4df-kube-api-access-mhk7v" (OuterVolumeSpecName: "kube-api-access-mhk7v") pod "9fac4a7c-3089-4b7b-a111-fd10fa79f4df" (UID: "9fac4a7c-3089-4b7b-a111-fd10fa79f4df"). InnerVolumeSpecName "kube-api-access-mhk7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:00:21.143195 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:21.143125 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fac4a7c-3089-4b7b-a111-fd10fa79f4df-plugins-config-volume\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:00:21.143195 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:21.143149 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhk7v\" (UniqueName: \"kubernetes.io/projected/9fac4a7c-3089-4b7b-a111-fd10fa79f4df-kube-api-access-mhk7v\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:00:21.170208 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:21.170178 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c816a64d-aab1-44a1-8209-9548164db02f" path="/var/lib/kubelet/pods/c816a64d-aab1-44a1-8209-9548164db02f/volumes" Sep 10 16:00:21.462346 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:21.462261 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:21.557163 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:21.557124 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:21.704527 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:21.704497 2576 generic.go:356] "Generic (PLEG): container finished" podID="2c12953a-5e35-480c-8395-95cfd2b6c98c" containerID="f41af2524eca0dbc52578c220a12c219e50d1fb87bfa69962fe85724457be137" exitCode=0 Sep 10 16:00:21.704701 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:21.704539 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-678f625f-dbcf47945-9c9kv" Sep 10 16:00:21.704701 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:21.704593 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-678f625f-dbcf47945-9c9kv" event={"ID":"2c12953a-5e35-480c-8395-95cfd2b6c98c","Type":"ContainerDied","Data":"f41af2524eca0dbc52578c220a12c219e50d1fb87bfa69962fe85724457be137"} Sep 10 16:00:21.704701 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:21.704654 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-678f625f-dbcf47945-9c9kv" event={"ID":"2c12953a-5e35-480c-8395-95cfd2b6c98c","Type":"ContainerDied","Data":"5bd79668036c7427014d95d3c9fc8ad6a9d6c2250d4c323856fd5ffa429d48cd"} Sep 10 16:00:21.704701 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:21.704685 2576 scope.go:117] "RemoveContainer" containerID="f41af2524eca0dbc52578c220a12c219e50d1fb87bfa69962fe85724457be137" Sep 10 16:00:21.706175 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:21.706155 2576 generic.go:356] "Generic (PLEG): container finished" podID="9fac4a7c-3089-4b7b-a111-fd10fa79f4df" containerID="3dc2dc56434fed0a3b27a515bda42d49ce8e343f23f3b2298fc476a15afba9c9" exitCode=0 Sep 10 16:00:21.706289 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:21.706199 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-678f625f-59c7bc5874-trjv6" event={"ID":"9fac4a7c-3089-4b7b-a111-fd10fa79f4df","Type":"ContainerDied","Data":"3dc2dc56434fed0a3b27a515bda42d49ce8e343f23f3b2298fc476a15afba9c9"} Sep 10 16:00:21.706289 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:21.706220 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-678f625f-59c7bc5874-trjv6" Sep 10 16:00:21.706289 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:21.706241 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-678f625f-59c7bc5874-trjv6" event={"ID":"9fac4a7c-3089-4b7b-a111-fd10fa79f4df","Type":"ContainerDied","Data":"3cacbac7d3c7b35268dfa422c6d4f50d2136303cf1bb6ceaf99fbf3c391b3c02"} Sep 10 16:00:21.713722 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:21.713693 2576 scope.go:117] "RemoveContainer" containerID="f41af2524eca0dbc52578c220a12c219e50d1fb87bfa69962fe85724457be137" Sep 10 16:00:21.713998 ip-10-0-139-236 kubenswrapper[2576]: E0910 16:00:21.713979 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f41af2524eca0dbc52578c220a12c219e50d1fb87bfa69962fe85724457be137\": container with ID starting with f41af2524eca0dbc52578c220a12c219e50d1fb87bfa69962fe85724457be137 not found: ID does not exist" containerID="f41af2524eca0dbc52578c220a12c219e50d1fb87bfa69962fe85724457be137" Sep 10 16:00:21.714052 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:21.714006 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f41af2524eca0dbc52578c220a12c219e50d1fb87bfa69962fe85724457be137"} err="failed to get container status \"f41af2524eca0dbc52578c220a12c219e50d1fb87bfa69962fe85724457be137\": rpc error: code = NotFound desc = could not find container \"f41af2524eca0dbc52578c220a12c219e50d1fb87bfa69962fe85724457be137\": container with ID starting with f41af2524eca0dbc52578c220a12c219e50d1fb87bfa69962fe85724457be137 not found: ID does not exist" Sep 10 16:00:21.714052 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:21.714022 2576 scope.go:117] "RemoveContainer" containerID="3dc2dc56434fed0a3b27a515bda42d49ce8e343f23f3b2298fc476a15afba9c9" Sep 10 16:00:21.721717 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:21.721699 2576 scope.go:117] "RemoveContainer" containerID="3dc2dc56434fed0a3b27a515bda42d49ce8e343f23f3b2298fc476a15afba9c9" Sep 10 16:00:21.722017 ip-10-0-139-236 kubenswrapper[2576]: E0910 16:00:21.722001 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dc2dc56434fed0a3b27a515bda42d49ce8e343f23f3b2298fc476a15afba9c9\": container with ID starting with 3dc2dc56434fed0a3b27a515bda42d49ce8e343f23f3b2298fc476a15afba9c9 not found: ID does not exist" containerID="3dc2dc56434fed0a3b27a515bda42d49ce8e343f23f3b2298fc476a15afba9c9" Sep 10 16:00:21.722062 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:21.722023 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dc2dc56434fed0a3b27a515bda42d49ce8e343f23f3b2298fc476a15afba9c9"} err="failed to get container status \"3dc2dc56434fed0a3b27a515bda42d49ce8e343f23f3b2298fc476a15afba9c9\": rpc error: code = NotFound desc = could not find container \"3dc2dc56434fed0a3b27a515bda42d49ce8e343f23f3b2298fc476a15afba9c9\": container with ID starting with 3dc2dc56434fed0a3b27a515bda42d49ce8e343f23f3b2298fc476a15afba9c9 not found: ID does not exist" Sep 10 16:00:21.729250 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:21.729228 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-678f625f-dbcf47945-9c9kv"] Sep 10 16:00:21.731483 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:21.731465 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-678f625f-dbcf47945-9c9kv"] Sep 10 16:00:21.747305 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:21.747280 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-678f625f-59c7bc5874-trjv6"] Sep 10 16:00:21.748971 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:21.748953 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-678f625f-59c7bc5874-trjv6"] Sep 10 16:00:22.672964 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:22.672927 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:23.169290 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:23.169254 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c12953a-5e35-480c-8395-95cfd2b6c98c" path="/var/lib/kubelet/pods/2c12953a-5e35-480c-8395-95cfd2b6c98c/volumes" Sep 10 16:00:23.169604 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:23.169590 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fac4a7c-3089-4b7b-a111-fd10fa79f4df" path="/var/lib/kubelet/pods/9fac4a7c-3089-4b7b-a111-fd10fa79f4df/volumes" Sep 10 16:00:24.099030 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:24.098994 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 16:00:24.729942 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:24.729905 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:25.129605 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:25.129555 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:25.361474 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:25.361437 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:26.297889 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:26.297853 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:27.985056 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:27.985013 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:28.242606 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:28.242512 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj"] Sep 10 16:00:28.242836 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:28.242810 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj" podUID="1d500885-c94a-4f5e-b9da-61e310e6ad7c" containerName="istio-proxy" containerID="cri-o://23731ea95d12bc3094b647e16659b0f39e2c11305526c58f80daf785180b2e34" gracePeriod=30 Sep 10 16:00:28.359779 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:28.359718 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:28.441520 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:28.441462 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:28.612718 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:28.612685 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:28.741402 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:28.741365 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:28.843639 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:28.843589 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:28.944581 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:28.944479 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:29.578504 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:29.578466 2576 patch_prober.go:28] interesting pod/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 10 16:00:29.578922 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:29.578526 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj" podUID="1d500885-c94a-4f5e-b9da-61e310e6ad7c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 16:00:29.991294 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:29.991207 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:30.130273 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:30.130229 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:30.225351 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:30.225317 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:30.580838 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:30.580802 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:30.675339 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:30.675306 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-67df2b-6cbf8787ff-rdh76"] Sep 10 16:00:30.675683 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:30.675671 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c816a64d-aab1-44a1-8209-9548164db02f" containerName="cleanup" Sep 10 16:00:30.675727 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:30.675686 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c816a64d-aab1-44a1-8209-9548164db02f" containerName="cleanup" Sep 10 16:00:30.675727 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:30.675709 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fac4a7c-3089-4b7b-a111-fd10fa79f4df" containerName="payload-pre-processing" Sep 10 16:00:30.675727 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:30.675715 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fac4a7c-3089-4b7b-a111-fd10fa79f4df" containerName="payload-pre-processing" Sep 10 16:00:30.675727 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:30.675722 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c12953a-5e35-480c-8395-95cfd2b6c98c" containerName="payload-processing" Sep 10 16:00:30.675727 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:30.675743 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c12953a-5e35-480c-8395-95cfd2b6c98c" containerName="payload-processing" Sep 10 16:00:30.675916 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:30.675830 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c816a64d-aab1-44a1-8209-9548164db02f" containerName="cleanup" Sep 10 16:00:30.675916 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:30.675839 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c12953a-5e35-480c-8395-95cfd2b6c98c" containerName="payload-processing" Sep 10 16:00:30.675916 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:30.675848 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fac4a7c-3089-4b7b-a111-fd10fa79f4df" containerName="payload-pre-processing" Sep 10 16:00:30.678616 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:30.678596 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-67df2b-6cbf8787ff-rdh76" Sep 10 16:00:30.681138 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:30.681117 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-67df2b\"" Sep 10 16:00:30.691147 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:30.691127 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-67df2b-6cbf8787ff-rdh76"] Sep 10 16:00:30.709446 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:30.709412 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:30.740992 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:30.740956 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dec1a26-3316-4023-ba38-d339a73e4549-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-67df2b-6cbf8787ff-rdh76\" (UID: \"1dec1a26-3316-4023-ba38-d339a73e4549\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-67df2b-6cbf8787ff-rdh76" Sep 10 16:00:30.741178 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:30.741071 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6wqk\" (UniqueName: \"kubernetes.io/projected/1dec1a26-3316-4023-ba38-d339a73e4549-kube-api-access-j6wqk\") pod \"payload-processing-e2e-shared-b-w3-67df2b-6cbf8787ff-rdh76\" (UID: \"1dec1a26-3316-4023-ba38-d339a73e4549\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-67df2b-6cbf8787ff-rdh76" Sep 10 16:00:30.842359 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:30.842278 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6wqk\" (UniqueName: \"kubernetes.io/projected/1dec1a26-3316-4023-ba38-d339a73e4549-kube-api-access-j6wqk\") pod \"payload-processing-e2e-shared-b-w3-67df2b-6cbf8787ff-rdh76\" (UID: \"1dec1a26-3316-4023-ba38-d339a73e4549\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-67df2b-6cbf8787ff-rdh76" Sep 10 16:00:30.842526 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:30.842388 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dec1a26-3316-4023-ba38-d339a73e4549-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-67df2b-6cbf8787ff-rdh76\" (UID: \"1dec1a26-3316-4023-ba38-d339a73e4549\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-67df2b-6cbf8787ff-rdh76" Sep 10 16:00:30.843127 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:30.843108 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dec1a26-3316-4023-ba38-d339a73e4549-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-67df2b-6cbf8787ff-rdh76\" (UID: \"1dec1a26-3316-4023-ba38-d339a73e4549\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-67df2b-6cbf8787ff-rdh76" Sep 10 16:00:30.856040 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:30.856011 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6wqk\" (UniqueName: \"kubernetes.io/projected/1dec1a26-3316-4023-ba38-d339a73e4549-kube-api-access-j6wqk\") pod \"payload-processing-e2e-shared-b-w3-67df2b-6cbf8787ff-rdh76\" (UID: \"1dec1a26-3316-4023-ba38-d339a73e4549\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-67df2b-6cbf8787ff-rdh76" Sep 10 16:00:30.988298 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:30.988268 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-67df2b-6cbf8787ff-rdh76" Sep 10 16:00:31.266980 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:31.263240 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-67df2b-6cbf8787ff-rdh76"] Sep 10 16:00:31.763941 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:31.763898 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-67df2b-6cbf8787ff-rdh76" event={"ID":"1dec1a26-3316-4023-ba38-d339a73e4549","Type":"ContainerStarted","Data":"a90dc05c01038c6ab9386ab4454c2ecb4fe64dca8835c3509dd5f26f215f8b6a"} Sep 10 16:00:31.764342 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:31.763966 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-67df2b-6cbf8787ff-rdh76" event={"ID":"1dec1a26-3316-4023-ba38-d339a73e4549","Type":"ContainerStarted","Data":"c75923137ab24fcda23899e563d34eaa7c9491cc792046e06677ed02a7955d36"} Sep 10 16:00:31.764342 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:31.763997 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-67df2b-6cbf8787ff-rdh76" Sep 10 16:00:31.786869 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:31.786811 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-67df2b-6cbf8787ff-rdh76" podStartSLOduration=1.786789711 podStartE2EDuration="1.786789711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:00:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:00:31.78402177 +0000 UTC m=+1109.137540945" watchObservedRunningTime="2026-09-10 16:00:31.786789711 +0000 UTC m=+1109.140308874" Sep 10 16:00:33.507083 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:33.507001 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj" Sep 10 16:00:33.568457 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:33.568425 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d500885-c94a-4f5e-b9da-61e310e6ad7c-istio-data\") pod \"1d500885-c94a-4f5e-b9da-61e310e6ad7c\" (UID: \"1d500885-c94a-4f5e-b9da-61e310e6ad7c\") " Sep 10 16:00:33.568457 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:33.568459 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d500885-c94a-4f5e-b9da-61e310e6ad7c-istio-token\") pod \"1d500885-c94a-4f5e-b9da-61e310e6ad7c\" (UID: \"1d500885-c94a-4f5e-b9da-61e310e6ad7c\") " Sep 10 16:00:33.568705 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:33.568488 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d500885-c94a-4f5e-b9da-61e310e6ad7c-istio-envoy\") pod \"1d500885-c94a-4f5e-b9da-61e310e6ad7c\" (UID: \"1d500885-c94a-4f5e-b9da-61e310e6ad7c\") " Sep 10 16:00:33.568705 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:33.568507 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d500885-c94a-4f5e-b9da-61e310e6ad7c-istio-podinfo\") pod \"1d500885-c94a-4f5e-b9da-61e310e6ad7c\" (UID: \"1d500885-c94a-4f5e-b9da-61e310e6ad7c\") " Sep 10 16:00:33.568705 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:33.568527 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d500885-c94a-4f5e-b9da-61e310e6ad7c-istiod-ca-cert\") pod \"1d500885-c94a-4f5e-b9da-61e310e6ad7c\" (UID: \"1d500885-c94a-4f5e-b9da-61e310e6ad7c\") " Sep 10 16:00:33.568705 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:33.568557 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fnsfs\" (UniqueName: \"kubernetes.io/projected/1d500885-c94a-4f5e-b9da-61e310e6ad7c-kube-api-access-fnsfs\") pod \"1d500885-c94a-4f5e-b9da-61e310e6ad7c\" (UID: \"1d500885-c94a-4f5e-b9da-61e310e6ad7c\") " Sep 10 16:00:33.568705 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:33.568593 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d500885-c94a-4f5e-b9da-61e310e6ad7c-workload-certs\") pod \"1d500885-c94a-4f5e-b9da-61e310e6ad7c\" (UID: \"1d500885-c94a-4f5e-b9da-61e310e6ad7c\") " Sep 10 16:00:33.568705 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:33.568642 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d500885-c94a-4f5e-b9da-61e310e6ad7c-workload-socket\") pod \"1d500885-c94a-4f5e-b9da-61e310e6ad7c\" (UID: \"1d500885-c94a-4f5e-b9da-61e310e6ad7c\") " Sep 10 16:00:33.568705 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:33.568686 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d500885-c94a-4f5e-b9da-61e310e6ad7c-credential-socket\") pod \"1d500885-c94a-4f5e-b9da-61e310e6ad7c\" (UID: \"1d500885-c94a-4f5e-b9da-61e310e6ad7c\") " Sep 10 16:00:33.569095 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:33.568780 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d500885-c94a-4f5e-b9da-61e310e6ad7c-istio-data" (OuterVolumeSpecName: "istio-data") pod "1d500885-c94a-4f5e-b9da-61e310e6ad7c" (UID: "1d500885-c94a-4f5e-b9da-61e310e6ad7c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:00:33.569095 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:33.568982 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d500885-c94a-4f5e-b9da-61e310e6ad7c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1d500885-c94a-4f5e-b9da-61e310e6ad7c" (UID: "1d500885-c94a-4f5e-b9da-61e310e6ad7c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:00:33.569095 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:33.569017 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d500885-c94a-4f5e-b9da-61e310e6ad7c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1d500885-c94a-4f5e-b9da-61e310e6ad7c" (UID: "1d500885-c94a-4f5e-b9da-61e310e6ad7c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:00:33.569095 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:33.569034 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d500885-c94a-4f5e-b9da-61e310e6ad7c-istio-data\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:00:33.569095 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:33.568967 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d500885-c94a-4f5e-b9da-61e310e6ad7c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1d500885-c94a-4f5e-b9da-61e310e6ad7c" (UID: "1d500885-c94a-4f5e-b9da-61e310e6ad7c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:00:33.569524 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:33.569152 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d500885-c94a-4f5e-b9da-61e310e6ad7c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1d500885-c94a-4f5e-b9da-61e310e6ad7c" (UID: "1d500885-c94a-4f5e-b9da-61e310e6ad7c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:00:33.571164 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:33.571133 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d500885-c94a-4f5e-b9da-61e310e6ad7c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1d500885-c94a-4f5e-b9da-61e310e6ad7c" (UID: "1d500885-c94a-4f5e-b9da-61e310e6ad7c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 16:00:33.571287 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:33.571201 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d500885-c94a-4f5e-b9da-61e310e6ad7c-kube-api-access-fnsfs" (OuterVolumeSpecName: "kube-api-access-fnsfs") pod "1d500885-c94a-4f5e-b9da-61e310e6ad7c" (UID: "1d500885-c94a-4f5e-b9da-61e310e6ad7c"). InnerVolumeSpecName "kube-api-access-fnsfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:00:33.571610 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:33.571587 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d500885-c94a-4f5e-b9da-61e310e6ad7c-istio-token" (OuterVolumeSpecName: "istio-token") pod "1d500885-c94a-4f5e-b9da-61e310e6ad7c" (UID: "1d500885-c94a-4f5e-b9da-61e310e6ad7c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:00:33.571610 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:33.571590 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d500885-c94a-4f5e-b9da-61e310e6ad7c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1d500885-c94a-4f5e-b9da-61e310e6ad7c" (UID: "1d500885-c94a-4f5e-b9da-61e310e6ad7c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:00:33.596902 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:33.596864 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 16:00:33.669851 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:33.669814 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d500885-c94a-4f5e-b9da-61e310e6ad7c-credential-socket\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:00:33.669851 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:33.669846 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d500885-c94a-4f5e-b9da-61e310e6ad7c-istio-token\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:00:33.669851 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:33.669859 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d500885-c94a-4f5e-b9da-61e310e6ad7c-istio-envoy\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:00:33.670122 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:33.669870 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d500885-c94a-4f5e-b9da-61e310e6ad7c-istio-podinfo\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:00:33.670122 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:33.669879 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d500885-c94a-4f5e-b9da-61e310e6ad7c-istiod-ca-cert\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:00:33.670122 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:33.669888 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnsfs\" (UniqueName: \"kubernetes.io/projected/1d500885-c94a-4f5e-b9da-61e310e6ad7c-kube-api-access-fnsfs\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:00:33.670122 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:33.669898 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d500885-c94a-4f5e-b9da-61e310e6ad7c-workload-certs\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:00:33.670122 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:33.669909 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d500885-c94a-4f5e-b9da-61e310e6ad7c-workload-socket\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:00:33.777010 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:33.776917 2576 generic.go:356] "Generic (PLEG): container finished" podID="1d500885-c94a-4f5e-b9da-61e310e6ad7c" containerID="23731ea95d12bc3094b647e16659b0f39e2c11305526c58f80daf785180b2e34" exitCode=0 Sep 10 16:00:33.777010 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:33.776970 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj" Sep 10 16:00:33.777010 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:33.776993 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj" event={"ID":"1d500885-c94a-4f5e-b9da-61e310e6ad7c","Type":"ContainerDied","Data":"23731ea95d12bc3094b647e16659b0f39e2c11305526c58f80daf785180b2e34"} Sep 10 16:00:33.777254 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:33.777052 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj" event={"ID":"1d500885-c94a-4f5e-b9da-61e310e6ad7c","Type":"ContainerDied","Data":"2ee08553edb96113f418ff894a3b89229a26ca952859e18ef8b7164109a0715a"} Sep 10 16:00:33.777254 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:33.777075 2576 scope.go:117] "RemoveContainer" containerID="23731ea95d12bc3094b647e16659b0f39e2c11305526c58f80daf785180b2e34" Sep 10 16:00:33.787010 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:33.786992 2576 scope.go:117] "RemoveContainer" containerID="3513b9c880370e635603a4f9290c86b5a151b96a9b05f94148a8b59cfe99707d" Sep 10 16:00:33.795225 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:33.795207 2576 scope.go:117] "RemoveContainer" containerID="23731ea95d12bc3094b647e16659b0f39e2c11305526c58f80daf785180b2e34" Sep 10 16:00:33.795504 ip-10-0-139-236 kubenswrapper[2576]: E0910 16:00:33.795487 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23731ea95d12bc3094b647e16659b0f39e2c11305526c58f80daf785180b2e34\": container with ID starting with 23731ea95d12bc3094b647e16659b0f39e2c11305526c58f80daf785180b2e34 not found: ID does not exist" containerID="23731ea95d12bc3094b647e16659b0f39e2c11305526c58f80daf785180b2e34" Sep 10 16:00:33.795580 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:33.795516 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23731ea95d12bc3094b647e16659b0f39e2c11305526c58f80daf785180b2e34"} err="failed to get container status \"23731ea95d12bc3094b647e16659b0f39e2c11305526c58f80daf785180b2e34\": rpc error: code = NotFound desc = could not find container \"23731ea95d12bc3094b647e16659b0f39e2c11305526c58f80daf785180b2e34\": container with ID starting with 23731ea95d12bc3094b647e16659b0f39e2c11305526c58f80daf785180b2e34 not found: ID does not exist" Sep 10 16:00:33.795580 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:33.795540 2576 scope.go:117] "RemoveContainer" containerID="3513b9c880370e635603a4f9290c86b5a151b96a9b05f94148a8b59cfe99707d" Sep 10 16:00:33.795865 ip-10-0-139-236 kubenswrapper[2576]: E0910 16:00:33.795847 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3513b9c880370e635603a4f9290c86b5a151b96a9b05f94148a8b59cfe99707d\": container with ID starting with 3513b9c880370e635603a4f9290c86b5a151b96a9b05f94148a8b59cfe99707d not found: ID does not exist" containerID="3513b9c880370e635603a4f9290c86b5a151b96a9b05f94148a8b59cfe99707d" Sep 10 16:00:33.795912 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:33.795870 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3513b9c880370e635603a4f9290c86b5a151b96a9b05f94148a8b59cfe99707d"} err="failed to get container status \"3513b9c880370e635603a4f9290c86b5a151b96a9b05f94148a8b59cfe99707d\": rpc error: code = NotFound desc = could not find container \"3513b9c880370e635603a4f9290c86b5a151b96a9b05f94148a8b59cfe99707d\": container with ID starting with 3513b9c880370e635603a4f9290c86b5a151b96a9b05f94148a8b59cfe99707d not found: ID does not exist" Sep 10 16:00:33.802237 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:33.802210 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj"] Sep 10 16:00:33.810381 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:33.810356 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-678f625f-openshift-default-69b54854cb-sxspj"] Sep 10 16:00:35.169528 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:35.169492 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d500885-c94a-4f5e-b9da-61e310e6ad7c" path="/var/lib/kubelet/pods/1d500885-c94a-4f5e-b9da-61e310e6ad7c/volumes" Sep 10 16:00:35.293714 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:35.293680 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:35.394946 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:35.394909 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:35.491519 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:35.491436 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:35.663756 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:35.663702 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:41.613005 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:41.612967 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:41.713651 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:41.713612 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:42.640919 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:42.640886 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:42.771921 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:42.771889 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-67df2b-6cbf8787ff-rdh76" Sep 10 16:00:42.980515 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:42.980437 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4"] Sep 10 16:00:42.980911 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:42.980888 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d500885-c94a-4f5e-b9da-61e310e6ad7c" containerName="istio-proxy" Sep 10 16:00:42.980911 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:42.980905 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d500885-c94a-4f5e-b9da-61e310e6ad7c" containerName="istio-proxy" Sep 10 16:00:42.980911 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:42.980913 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d500885-c94a-4f5e-b9da-61e310e6ad7c" containerName="istio-proxy" Sep 10 16:00:42.980911 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:42.980918 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d500885-c94a-4f5e-b9da-61e310e6ad7c" containerName="istio-proxy" Sep 10 16:00:42.981131 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:42.980978 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d500885-c94a-4f5e-b9da-61e310e6ad7c" containerName="istio-proxy" Sep 10 16:00:42.981131 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:42.980986 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d500885-c94a-4f5e-b9da-61e310e6ad7c" containerName="istio-proxy" Sep 10 16:00:42.985804 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:42.985785 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4" Sep 10 16:00:42.992966 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:42.992941 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-a361a9-openshift-default-dockercfg-9j96z\"" Sep 10 16:00:43.002273 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:43.002246 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4"] Sep 10 16:00:43.034105 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:43.034070 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:43.056132 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:43.056109 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/490fdd5a-363b-4314-92a1-23bc1f896c53-credential-socket\") pod \"e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4\" (UID: \"490fdd5a-363b-4314-92a1-23bc1f896c53\") " pod="openshift-ingress/e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4" Sep 10 16:00:43.056303 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:43.056145 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/490fdd5a-363b-4314-92a1-23bc1f896c53-istio-data\") pod \"e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4\" (UID: \"490fdd5a-363b-4314-92a1-23bc1f896c53\") " pod="openshift-ingress/e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4" Sep 10 16:00:43.056303 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:43.056176 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/490fdd5a-363b-4314-92a1-23bc1f896c53-workload-socket\") pod \"e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4\" (UID: \"490fdd5a-363b-4314-92a1-23bc1f896c53\") " pod="openshift-ingress/e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4" Sep 10 16:00:43.056425 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:43.056306 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rgkw\" (UniqueName: \"kubernetes.io/projected/490fdd5a-363b-4314-92a1-23bc1f896c53-kube-api-access-2rgkw\") pod \"e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4\" (UID: \"490fdd5a-363b-4314-92a1-23bc1f896c53\") " pod="openshift-ingress/e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4" Sep 10 16:00:43.056425 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:43.056344 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/490fdd5a-363b-4314-92a1-23bc1f896c53-workload-certs\") pod \"e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4\" (UID: \"490fdd5a-363b-4314-92a1-23bc1f896c53\") " pod="openshift-ingress/e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4" Sep 10 16:00:43.056425 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:43.056400 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/490fdd5a-363b-4314-92a1-23bc1f896c53-istio-podinfo\") pod \"e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4\" (UID: \"490fdd5a-363b-4314-92a1-23bc1f896c53\") " pod="openshift-ingress/e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4" Sep 10 16:00:43.056523 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:43.056458 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/490fdd5a-363b-4314-92a1-23bc1f896c53-istio-envoy\") pod \"e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4\" (UID: \"490fdd5a-363b-4314-92a1-23bc1f896c53\") " pod="openshift-ingress/e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4" Sep 10 16:00:43.056523 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:43.056477 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/490fdd5a-363b-4314-92a1-23bc1f896c53-istio-token\") pod \"e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4\" (UID: \"490fdd5a-363b-4314-92a1-23bc1f896c53\") " pod="openshift-ingress/e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4" Sep 10 16:00:43.056523 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:43.056491 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/490fdd5a-363b-4314-92a1-23bc1f896c53-istiod-ca-cert\") pod \"e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4\" (UID: \"490fdd5a-363b-4314-92a1-23bc1f896c53\") " pod="openshift-ingress/e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4" Sep 10 16:00:43.156944 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:43.156911 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rgkw\" (UniqueName: \"kubernetes.io/projected/490fdd5a-363b-4314-92a1-23bc1f896c53-kube-api-access-2rgkw\") pod \"e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4\" (UID: \"490fdd5a-363b-4314-92a1-23bc1f896c53\") " pod="openshift-ingress/e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4" Sep 10 16:00:43.156944 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:43.156946 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/490fdd5a-363b-4314-92a1-23bc1f896c53-workload-certs\") pod \"e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4\" (UID: \"490fdd5a-363b-4314-92a1-23bc1f896c53\") " pod="openshift-ingress/e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4" Sep 10 16:00:43.157203 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:43.156970 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/490fdd5a-363b-4314-92a1-23bc1f896c53-istio-podinfo\") pod \"e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4\" (UID: \"490fdd5a-363b-4314-92a1-23bc1f896c53\") " pod="openshift-ingress/e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4" Sep 10 16:00:43.157203 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:43.157004 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/490fdd5a-363b-4314-92a1-23bc1f896c53-istio-envoy\") pod \"e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4\" (UID: \"490fdd5a-363b-4314-92a1-23bc1f896c53\") " pod="openshift-ingress/e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4" Sep 10 16:00:43.157203 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:43.157020 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/490fdd5a-363b-4314-92a1-23bc1f896c53-istio-token\") pod \"e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4\" (UID: \"490fdd5a-363b-4314-92a1-23bc1f896c53\") " pod="openshift-ingress/e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4" Sep 10 16:00:43.157203 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:43.157036 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/490fdd5a-363b-4314-92a1-23bc1f896c53-istiod-ca-cert\") pod \"e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4\" (UID: \"490fdd5a-363b-4314-92a1-23bc1f896c53\") " pod="openshift-ingress/e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4" Sep 10 16:00:43.157203 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:43.157062 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/490fdd5a-363b-4314-92a1-23bc1f896c53-credential-socket\") pod \"e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4\" (UID: \"490fdd5a-363b-4314-92a1-23bc1f896c53\") " pod="openshift-ingress/e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4" Sep 10 16:00:43.157203 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:43.157090 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/490fdd5a-363b-4314-92a1-23bc1f896c53-istio-data\") pod \"e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4\" (UID: \"490fdd5a-363b-4314-92a1-23bc1f896c53\") " pod="openshift-ingress/e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4" Sep 10 16:00:43.157203 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:43.157133 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/490fdd5a-363b-4314-92a1-23bc1f896c53-workload-socket\") pod \"e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4\" (UID: \"490fdd5a-363b-4314-92a1-23bc1f896c53\") " pod="openshift-ingress/e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4" Sep 10 16:00:43.157591 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:43.157412 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/490fdd5a-363b-4314-92a1-23bc1f896c53-workload-certs\") pod \"e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4\" (UID: \"490fdd5a-363b-4314-92a1-23bc1f896c53\") " pod="openshift-ingress/e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4" Sep 10 16:00:43.157591 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:43.157464 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/490fdd5a-363b-4314-92a1-23bc1f896c53-credential-socket\") pod \"e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4\" (UID: \"490fdd5a-363b-4314-92a1-23bc1f896c53\") " pod="openshift-ingress/e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4" Sep 10 16:00:43.157701 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:43.157662 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/490fdd5a-363b-4314-92a1-23bc1f896c53-istio-data\") pod \"e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4\" (UID: \"490fdd5a-363b-4314-92a1-23bc1f896c53\") " pod="openshift-ingress/e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4" Sep 10 16:00:43.157799 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:43.157706 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/490fdd5a-363b-4314-92a1-23bc1f896c53-workload-socket\") pod \"e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4\" (UID: \"490fdd5a-363b-4314-92a1-23bc1f896c53\") " pod="openshift-ingress/e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4" Sep 10 16:00:43.157849 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:43.157799 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/490fdd5a-363b-4314-92a1-23bc1f896c53-istiod-ca-cert\") pod \"e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4\" (UID: \"490fdd5a-363b-4314-92a1-23bc1f896c53\") " pod="openshift-ingress/e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4" Sep 10 16:00:43.159401 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:43.159381 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/490fdd5a-363b-4314-92a1-23bc1f896c53-istio-envoy\") pod \"e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4\" (UID: \"490fdd5a-363b-4314-92a1-23bc1f896c53\") " pod="openshift-ingress/e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4" Sep 10 16:00:43.159622 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:43.159606 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/490fdd5a-363b-4314-92a1-23bc1f896c53-istio-podinfo\") pod \"e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4\" (UID: \"490fdd5a-363b-4314-92a1-23bc1f896c53\") " pod="openshift-ingress/e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4" Sep 10 16:00:43.174155 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:43.174124 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/490fdd5a-363b-4314-92a1-23bc1f896c53-istio-token\") pod \"e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4\" (UID: \"490fdd5a-363b-4314-92a1-23bc1f896c53\") " pod="openshift-ingress/e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4" Sep 10 16:00:43.175282 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:43.175265 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rgkw\" (UniqueName: \"kubernetes.io/projected/490fdd5a-363b-4314-92a1-23bc1f896c53-kube-api-access-2rgkw\") pod \"e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4\" (UID: \"490fdd5a-363b-4314-92a1-23bc1f896c53\") " pod="openshift-ingress/e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4" Sep 10 16:00:43.295598 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:43.295559 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4" Sep 10 16:00:43.505352 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:43.505318 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:43.580848 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:43.580822 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4"] Sep 10 16:00:43.582259 ip-10-0-139-236 kubenswrapper[2576]: W0910 16:00:43.582232 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod490fdd5a_363b_4314_92a1_23bc1f896c53.slice/crio-96e29b00d436cb688a063e4f7c2e046ad007399e9b980146999e1c3132c09bef WatchSource:0}: Error finding container 96e29b00d436cb688a063e4f7c2e046ad007399e9b980146999e1c3132c09bef: Status 404 returned error can't find the container with id 96e29b00d436cb688a063e4f7c2e046ad007399e9b980146999e1c3132c09bef Sep 10 16:00:43.584473 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:43.584440 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:00:43.584569 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:43.584519 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:00:43.584633 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:43.584567 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:00:43.834076 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:43.833990 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4" event={"ID":"490fdd5a-363b-4314-92a1-23bc1f896c53","Type":"ContainerStarted","Data":"cf55a4ae2cf420e2ce6cddc67073593fbf01228c61e9bd954ee0b0f82586f2ce"} Sep 10 16:00:43.834076 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:43.834048 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4" event={"ID":"490fdd5a-363b-4314-92a1-23bc1f896c53","Type":"ContainerStarted","Data":"96e29b00d436cb688a063e4f7c2e046ad007399e9b980146999e1c3132c09bef"} Sep 10 16:00:43.859360 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:43.859313 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4" podStartSLOduration=1.859293015 podStartE2EDuration="1.859293015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:00:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:00:43.857525604 +0000 UTC m=+1121.211044797" watchObservedRunningTime="2026-09-10 16:00:43.859293015 +0000 UTC m=+1121.212812179" Sep 10 16:00:44.296236 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:44.296209 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4" Sep 10 16:00:44.301554 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:44.301527 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4" Sep 10 16:00:44.839163 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:44.839136 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4" Sep 10 16:00:44.840026 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:44.840008 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4" Sep 10 16:00:46.338276 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:46.338233 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:46.436343 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:46.436312 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:47.302697 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:47.302662 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:47.506245 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:47.506207 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:47.663971 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:47.663884 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:48.194467 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:48.194431 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:48.311161 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:48.311122 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:48.391906 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:48.391874 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a361a9-7f976dfdc8-7hbtq"] Sep 10 16:00:48.395821 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:48.395795 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a361a9-7f976dfdc8-7hbtq" Sep 10 16:00:48.398281 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:48.398248 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-a361a9-dockercfg-v57lr\"" Sep 10 16:00:48.398281 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:48.398269 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-a361a9\"" Sep 10 16:00:48.411660 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:48.411631 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a361a9-7f976dfdc8-7hbtq"] Sep 10 16:00:48.497559 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:48.497461 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-a361a9-9f6d5f9c5-gs8hx"] Sep 10 16:00:48.501946 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:48.501920 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a361a9-9f6d5f9c5-gs8hx" Sep 10 16:00:48.511261 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:48.511211 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f4b64a3-888f-42b9-be57-c38f5c500873-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-a361a9-7f976dfdc8-7hbtq\" (UID: \"0f4b64a3-888f-42b9-be57-c38f5c500873\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a361a9-7f976dfdc8-7hbtq" Sep 10 16:00:48.511635 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:48.511273 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zcgn\" (UniqueName: \"kubernetes.io/projected/0f4b64a3-888f-42b9-be57-c38f5c500873-kube-api-access-7zcgn\") pod \"payload-pre-processing-e2e-shared-a-w2-a361a9-7f976dfdc8-7hbtq\" (UID: \"0f4b64a3-888f-42b9-be57-c38f5c500873\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a361a9-7f976dfdc8-7hbtq" Sep 10 16:00:48.516510 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:48.516485 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-a361a9-9f6d5f9c5-gs8hx"] Sep 10 16:00:48.612136 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:48.612097 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjjmk\" (UniqueName: \"kubernetes.io/projected/a79ea085-ab37-462b-aa56-81f09c2eb640-kube-api-access-kjjmk\") pod \"payload-processing-e2e-shared-a-w2-a361a9-9f6d5f9c5-gs8hx\" (UID: \"a79ea085-ab37-462b-aa56-81f09c2eb640\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a361a9-9f6d5f9c5-gs8hx" Sep 10 16:00:48.612311 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:48.612180 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f4b64a3-888f-42b9-be57-c38f5c500873-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-a361a9-7f976dfdc8-7hbtq\" (UID: \"0f4b64a3-888f-42b9-be57-c38f5c500873\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a361a9-7f976dfdc8-7hbtq" Sep 10 16:00:48.612311 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:48.612207 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a79ea085-ab37-462b-aa56-81f09c2eb640-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-a361a9-9f6d5f9c5-gs8hx\" (UID: \"a79ea085-ab37-462b-aa56-81f09c2eb640\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a361a9-9f6d5f9c5-gs8hx" Sep 10 16:00:48.612311 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:48.612231 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zcgn\" (UniqueName: \"kubernetes.io/projected/0f4b64a3-888f-42b9-be57-c38f5c500873-kube-api-access-7zcgn\") pod \"payload-pre-processing-e2e-shared-a-w2-a361a9-7f976dfdc8-7hbtq\" (UID: \"0f4b64a3-888f-42b9-be57-c38f5c500873\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a361a9-7f976dfdc8-7hbtq" Sep 10 16:00:48.612980 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:48.612959 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f4b64a3-888f-42b9-be57-c38f5c500873-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-a361a9-7f976dfdc8-7hbtq\" (UID: \"0f4b64a3-888f-42b9-be57-c38f5c500873\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a361a9-7f976dfdc8-7hbtq" Sep 10 16:00:48.623807 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:48.623774 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zcgn\" (UniqueName: \"kubernetes.io/projected/0f4b64a3-888f-42b9-be57-c38f5c500873-kube-api-access-7zcgn\") pod \"payload-pre-processing-e2e-shared-a-w2-a361a9-7f976dfdc8-7hbtq\" (UID: \"0f4b64a3-888f-42b9-be57-c38f5c500873\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a361a9-7f976dfdc8-7hbtq" Sep 10 16:00:48.706920 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:48.706886 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a361a9-7f976dfdc8-7hbtq" Sep 10 16:00:48.713375 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:48.713350 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a79ea085-ab37-462b-aa56-81f09c2eb640-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-a361a9-9f6d5f9c5-gs8hx\" (UID: \"a79ea085-ab37-462b-aa56-81f09c2eb640\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a361a9-9f6d5f9c5-gs8hx" Sep 10 16:00:48.713500 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:48.713410 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjjmk\" (UniqueName: \"kubernetes.io/projected/a79ea085-ab37-462b-aa56-81f09c2eb640-kube-api-access-kjjmk\") pod \"payload-processing-e2e-shared-a-w2-a361a9-9f6d5f9c5-gs8hx\" (UID: \"a79ea085-ab37-462b-aa56-81f09c2eb640\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a361a9-9f6d5f9c5-gs8hx" Sep 10 16:00:48.714048 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:48.714024 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a79ea085-ab37-462b-aa56-81f09c2eb640-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-a361a9-9f6d5f9c5-gs8hx\" (UID: \"a79ea085-ab37-462b-aa56-81f09c2eb640\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a361a9-9f6d5f9c5-gs8hx" Sep 10 16:00:48.725659 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:48.725631 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjjmk\" (UniqueName: \"kubernetes.io/projected/a79ea085-ab37-462b-aa56-81f09c2eb640-kube-api-access-kjjmk\") pod \"payload-processing-e2e-shared-a-w2-a361a9-9f6d5f9c5-gs8hx\" (UID: \"a79ea085-ab37-462b-aa56-81f09c2eb640\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a361a9-9f6d5f9c5-gs8hx" Sep 10 16:00:48.817043 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:48.817011 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a361a9-9f6d5f9c5-gs8hx" Sep 10 16:00:48.986385 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:48.986359 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a361a9-7f976dfdc8-7hbtq"] Sep 10 16:00:48.988194 ip-10-0-139-236 kubenswrapper[2576]: W0910 16:00:48.988167 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f4b64a3_888f_42b9_be57_c38f5c500873.slice/crio-16263cce5f8626be71b70c8a5c1278160a6f9b2ca4b839215e33932fbec0eeb1 WatchSource:0}: Error finding container 16263cce5f8626be71b70c8a5c1278160a6f9b2ca4b839215e33932fbec0eeb1: Status 404 returned error can't find the container with id 16263cce5f8626be71b70c8a5c1278160a6f9b2ca4b839215e33932fbec0eeb1 Sep 10 16:00:49.094409 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:49.094385 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-a361a9-9f6d5f9c5-gs8hx"] Sep 10 16:00:49.096203 ip-10-0-139-236 kubenswrapper[2576]: W0910 16:00:49.096176 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda79ea085_ab37_462b_aa56_81f09c2eb640.slice/crio-57ab541628409da6ae198d651c55c5e2f85eef583aec5952ff1ee0214d90c10d WatchSource:0}: Error finding container 57ab541628409da6ae198d651c55c5e2f85eef583aec5952ff1ee0214d90c10d: Status 404 returned error can't find the container with id 57ab541628409da6ae198d651c55c5e2f85eef583aec5952ff1ee0214d90c10d Sep 10 16:00:49.867045 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:49.867008 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a361a9-9f6d5f9c5-gs8hx" event={"ID":"a79ea085-ab37-462b-aa56-81f09c2eb640","Type":"ContainerStarted","Data":"a3e444c0fd5b49fb77d1856058b4f8ed24d8deaadddce92514245fd6758d9e18"} Sep 10 16:00:49.867496 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:49.867209 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a361a9-9f6d5f9c5-gs8hx" Sep 10 16:00:49.867496 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:49.867251 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a361a9-9f6d5f9c5-gs8hx" event={"ID":"a79ea085-ab37-462b-aa56-81f09c2eb640","Type":"ContainerStarted","Data":"57ab541628409da6ae198d651c55c5e2f85eef583aec5952ff1ee0214d90c10d"} Sep 10 16:00:49.868565 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:49.868540 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a361a9-7f976dfdc8-7hbtq" event={"ID":"0f4b64a3-888f-42b9-be57-c38f5c500873","Type":"ContainerStarted","Data":"ca906a7e2619c66ec5dae61e950ddc40cb66ba44a9248ea540581a02cfca80bf"} Sep 10 16:00:49.868670 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:49.868593 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a361a9-7f976dfdc8-7hbtq" Sep 10 16:00:49.868670 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:49.868617 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a361a9-7f976dfdc8-7hbtq" event={"ID":"0f4b64a3-888f-42b9-be57-c38f5c500873","Type":"ContainerStarted","Data":"16263cce5f8626be71b70c8a5c1278160a6f9b2ca4b839215e33932fbec0eeb1"} Sep 10 16:00:49.891032 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:49.890991 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a361a9-9f6d5f9c5-gs8hx" podStartSLOduration=1.890977396 podStartE2EDuration="1.890977396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:00:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:00:49.888441575 +0000 UTC m=+1127.241960752" watchObservedRunningTime="2026-09-10 16:00:49.890977396 +0000 UTC m=+1127.244496614" Sep 10 16:00:49.910084 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:49.910025 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a361a9-7f976dfdc8-7hbtq" podStartSLOduration=1.910010524 podStartE2EDuration="1.910010524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:00:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:00:49.907548486 +0000 UTC m=+1127.261067669" watchObservedRunningTime="2026-09-10 16:00:49.910010524 +0000 UTC m=+1127.263529722" Sep 10 16:00:51.123617 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:51.123582 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:53.690818 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:53.690775 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:54.090975 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:54.090938 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 16:00:54.246788 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:54.246728 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:54.510254 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:00:54.510163 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:01:00.875931 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:00.875896 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a361a9-7f976dfdc8-7hbtq" Sep 10 16:01:00.876310 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:00.875978 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a361a9-9f6d5f9c5-gs8hx" Sep 10 16:01:02.797993 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:02.797952 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:01:02.883293 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:02.883264 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:01:04.968017 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:04.967982 2576 generic.go:356] "Generic (PLEG): container finished" podID="490fdd5a-363b-4314-92a1-23bc1f896c53" containerID="cf55a4ae2cf420e2ce6cddc67073593fbf01228c61e9bd954ee0b0f82586f2ce" exitCode=0 Sep 10 16:01:04.968426 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:04.968057 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4" event={"ID":"490fdd5a-363b-4314-92a1-23bc1f896c53","Type":"ContainerDied","Data":"cf55a4ae2cf420e2ce6cddc67073593fbf01228c61e9bd954ee0b0f82586f2ce"} Sep 10 16:01:04.968514 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:04.968492 2576 scope.go:117] "RemoveContainer" containerID="cf55a4ae2cf420e2ce6cddc67073593fbf01228c61e9bd954ee0b0f82586f2ce" Sep 10 16:01:04.969574 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:04.969547 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:01:04.969625 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:04.969612 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:01:04.969661 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:04.969644 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:01:05.978652 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:05.978621 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4" event={"ID":"490fdd5a-363b-4314-92a1-23bc1f896c53","Type":"ContainerStarted","Data":"717f79f978dde50e9f15190a11eee80dfea663990c8b532df0c3ffd74eb434ef"} Sep 10 16:01:06.296480 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:06.296433 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4" Sep 10 16:01:06.296667 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:06.296475 2576 patch_prober.go:28] interesting pod/e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Sep 10 16:01:06.296667 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:06.296541 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4" podUID="490fdd5a-363b-4314-92a1-23bc1f896c53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Sep 10 16:01:07.314195 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:07.314161 2576 patch_prober.go:28] interesting pod/e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Sep 10 16:01:07.314848 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:07.314814 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4" podUID="490fdd5a-363b-4314-92a1-23bc1f896c53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Sep 10 16:01:08.296920 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:08.296885 2576 patch_prober.go:28] interesting pod/e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Sep 10 16:01:08.297213 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:08.297189 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4" podUID="490fdd5a-363b-4314-92a1-23bc1f896c53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Sep 10 16:01:08.714338 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:08.714244 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:01:09.296204 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:09.296167 2576 patch_prober.go:28] interesting pod/e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Sep 10 16:01:09.296509 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:09.296478 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4" podUID="490fdd5a-363b-4314-92a1-23bc1f896c53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Sep 10 16:01:10.296967 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:10.296933 2576 patch_prober.go:28] interesting pod/e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Sep 10 16:01:10.297554 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:10.297524 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4" podUID="490fdd5a-363b-4314-92a1-23bc1f896c53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Sep 10 16:01:10.833548 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:10.833496 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:01:11.182221 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:11.182136 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:01:11.300852 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:11.300827 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4" Sep 10 16:01:11.301264 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:11.301096 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4" Sep 10 16:01:11.301930 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:11.301910 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4" Sep 10 16:01:11.582117 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:11.582082 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:01:11.687207 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:11.687166 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:01:15.132850 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:15.132796 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:01:15.244566 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:15.244537 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:01:16.443546 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:16.443514 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:01:16.548256 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:16.548227 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:01:16.673545 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:16.673507 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:01:17.077579 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:17.077537 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:01:17.197756 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:17.197223 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:01:17.277619 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:17.277586 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-33451c-76474cff66-qwn9d"] Sep 10 16:01:17.282765 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:17.282719 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-33451c-76474cff66-qwn9d" Sep 10 16:01:17.286168 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:17.286147 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-33451c\"" Sep 10 16:01:17.286288 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:17.286198 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-33451c-dockercfg-n4wj7\"" Sep 10 16:01:17.295573 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:17.295549 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-33451c-76474cff66-qwn9d"] Sep 10 16:01:17.312010 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:17.311979 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93edc94e-43ca-4cf9-9716-b2478310d631-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-33451c-76474cff66-qwn9d\" (UID: \"93edc94e-43ca-4cf9-9716-b2478310d631\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-33451c-76474cff66-qwn9d" Sep 10 16:01:17.312232 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:17.312206 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz7cz\" (UniqueName: \"kubernetes.io/projected/93edc94e-43ca-4cf9-9716-b2478310d631-kube-api-access-lz7cz\") pod \"payload-pre-processing-e2e-shared-b-w2-33451c-76474cff66-qwn9d\" (UID: \"93edc94e-43ca-4cf9-9716-b2478310d631\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-33451c-76474cff66-qwn9d" Sep 10 16:01:17.365610 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:17.365521 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-33451c-857779b665-hdg6c"] Sep 10 16:01:17.369383 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:17.369356 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-33451c-857779b665-hdg6c" Sep 10 16:01:17.383188 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:17.383163 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-33451c-857779b665-hdg6c"] Sep 10 16:01:17.413450 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:17.413421 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93edc94e-43ca-4cf9-9716-b2478310d631-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-33451c-76474cff66-qwn9d\" (UID: \"93edc94e-43ca-4cf9-9716-b2478310d631\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-33451c-76474cff66-qwn9d" Sep 10 16:01:17.413594 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:17.413467 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz7cz\" (UniqueName: \"kubernetes.io/projected/93edc94e-43ca-4cf9-9716-b2478310d631-kube-api-access-lz7cz\") pod \"payload-pre-processing-e2e-shared-b-w2-33451c-76474cff66-qwn9d\" (UID: \"93edc94e-43ca-4cf9-9716-b2478310d631\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-33451c-76474cff66-qwn9d" Sep 10 16:01:17.413594 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:17.413514 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1b7117e-21cf-47b0-a845-0f7b7b78ecca-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-33451c-857779b665-hdg6c\" (UID: \"a1b7117e-21cf-47b0-a845-0f7b7b78ecca\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-33451c-857779b665-hdg6c" Sep 10 16:01:17.413594 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:17.413544 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cccm6\" (UniqueName: \"kubernetes.io/projected/a1b7117e-21cf-47b0-a845-0f7b7b78ecca-kube-api-access-cccm6\") pod \"payload-processing-e2e-shared-b-w2-33451c-857779b665-hdg6c\" (UID: \"a1b7117e-21cf-47b0-a845-0f7b7b78ecca\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-33451c-857779b665-hdg6c" Sep 10 16:01:17.414280 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:17.414256 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93edc94e-43ca-4cf9-9716-b2478310d631-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-33451c-76474cff66-qwn9d\" (UID: \"93edc94e-43ca-4cf9-9716-b2478310d631\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-33451c-76474cff66-qwn9d" Sep 10 16:01:17.427114 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:17.427090 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz7cz\" (UniqueName: \"kubernetes.io/projected/93edc94e-43ca-4cf9-9716-b2478310d631-kube-api-access-lz7cz\") pod \"payload-pre-processing-e2e-shared-b-w2-33451c-76474cff66-qwn9d\" (UID: \"93edc94e-43ca-4cf9-9716-b2478310d631\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-33451c-76474cff66-qwn9d" Sep 10 16:01:17.514657 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:17.514624 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1b7117e-21cf-47b0-a845-0f7b7b78ecca-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-33451c-857779b665-hdg6c\" (UID: \"a1b7117e-21cf-47b0-a845-0f7b7b78ecca\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-33451c-857779b665-hdg6c" Sep 10 16:01:17.515064 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:17.514692 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cccm6\" (UniqueName: \"kubernetes.io/projected/a1b7117e-21cf-47b0-a845-0f7b7b78ecca-kube-api-access-cccm6\") pod \"payload-processing-e2e-shared-b-w2-33451c-857779b665-hdg6c\" (UID: \"a1b7117e-21cf-47b0-a845-0f7b7b78ecca\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-33451c-857779b665-hdg6c" Sep 10 16:01:17.515397 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:17.515376 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1b7117e-21cf-47b0-a845-0f7b7b78ecca-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-33451c-857779b665-hdg6c\" (UID: \"a1b7117e-21cf-47b0-a845-0f7b7b78ecca\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-33451c-857779b665-hdg6c" Sep 10 16:01:17.528897 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:17.528871 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cccm6\" (UniqueName: \"kubernetes.io/projected/a1b7117e-21cf-47b0-a845-0f7b7b78ecca-kube-api-access-cccm6\") pod \"payload-processing-e2e-shared-b-w2-33451c-857779b665-hdg6c\" (UID: \"a1b7117e-21cf-47b0-a845-0f7b7b78ecca\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-33451c-857779b665-hdg6c" Sep 10 16:01:17.596521 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:17.596488 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-33451c-76474cff66-qwn9d" Sep 10 16:01:17.680324 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:17.680296 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-33451c-857779b665-hdg6c" Sep 10 16:01:17.878490 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:17.878468 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-33451c-76474cff66-qwn9d"] Sep 10 16:01:17.879930 ip-10-0-139-236 kubenswrapper[2576]: W0910 16:01:17.879905 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93edc94e_43ca_4cf9_9716_b2478310d631.slice/crio-b05c05bda589b5b998b544cb24e0af3c8b2f84f921f67319a0af116c20bf594c WatchSource:0}: Error finding container b05c05bda589b5b998b544cb24e0af3c8b2f84f921f67319a0af116c20bf594c: Status 404 returned error can't find the container with id b05c05bda589b5b998b544cb24e0af3c8b2f84f921f67319a0af116c20bf594c Sep 10 16:01:17.962779 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:17.962750 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-33451c-857779b665-hdg6c"] Sep 10 16:01:17.964226 ip-10-0-139-236 kubenswrapper[2576]: W0910 16:01:17.964202 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1b7117e_21cf_47b0_a845_0f7b7b78ecca.slice/crio-787f05e67faa6ac092fd04e7585eb1892df568c73a183c5bfe721630fbffb27a WatchSource:0}: Error finding container 787f05e67faa6ac092fd04e7585eb1892df568c73a183c5bfe721630fbffb27a: Status 404 returned error can't find the container with id 787f05e67faa6ac092fd04e7585eb1892df568c73a183c5bfe721630fbffb27a Sep 10 16:01:18.054150 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:18.054099 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-33451c-857779b665-hdg6c" event={"ID":"a1b7117e-21cf-47b0-a845-0f7b7b78ecca","Type":"ContainerStarted","Data":"8d8e4d1cab32d192a54e0a81096eb9601e91e314a4dea3f3bd989df6960059d5"} Sep 10 16:01:18.054298 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:18.054175 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-33451c-857779b665-hdg6c" event={"ID":"a1b7117e-21cf-47b0-a845-0f7b7b78ecca","Type":"ContainerStarted","Data":"787f05e67faa6ac092fd04e7585eb1892df568c73a183c5bfe721630fbffb27a"} Sep 10 16:01:18.054298 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:18.054204 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-33451c-857779b665-hdg6c" Sep 10 16:01:18.055830 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:18.055807 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-33451c-76474cff66-qwn9d" event={"ID":"93edc94e-43ca-4cf9-9716-b2478310d631","Type":"ContainerStarted","Data":"6851b686d21b22949d9ba682020ea07b976d2d82efda071f961d740408564e20"} Sep 10 16:01:18.055931 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:18.055846 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-33451c-76474cff66-qwn9d" event={"ID":"93edc94e-43ca-4cf9-9716-b2478310d631","Type":"ContainerStarted","Data":"b05c05bda589b5b998b544cb24e0af3c8b2f84f921f67319a0af116c20bf594c"} Sep 10 16:01:18.055981 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:18.055941 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-33451c-76474cff66-qwn9d" Sep 10 16:01:18.079835 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:18.079791 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-33451c-857779b665-hdg6c" podStartSLOduration=1.079776181 podStartE2EDuration="1.079776181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:01:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:01:18.07700604 +0000 UTC m=+1155.430525223" watchObservedRunningTime="2026-09-10 16:01:18.079776181 +0000 UTC m=+1155.433295344" Sep 10 16:01:18.092765 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:18.092687 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-33451c-76474cff66-qwn9d" podStartSLOduration=1.092666274 podStartE2EDuration="1.092666274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:01:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:01:18.092298841 +0000 UTC m=+1155.445818004" watchObservedRunningTime="2026-09-10 16:01:18.092666274 +0000 UTC m=+1155.446185440" Sep 10 16:01:19.353392 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:19.353363 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:01:21.895556 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:21.895512 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 16:01:28.153044 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:28.152955 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:01:28.220988 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:28.220960 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:01:29.064078 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:29.064046 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-33451c-76474cff66-qwn9d" Sep 10 16:01:29.064241 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:29.064199 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-33451c-857779b665-hdg6c" Sep 10 16:01:40.066175 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:40.066125 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:01:49.691149 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:49.691107 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:01:49.904315 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:49.904279 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:01:49.993440 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:49.993345 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:01:50.361479 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:50.361435 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:01:52.768668 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:52.768631 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:01:53.367659 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:53.367617 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:01:53.503718 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:53.503629 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:01:54.499536 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:54.499491 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:01:54.607299 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:54.607262 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:01:54.760324 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:54.760228 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:01:55.089640 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:55.089607 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:01:55.199364 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:55.199326 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:01:55.218777 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:55.218753 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0e6e0dcc-86c4d5f9db-r7bz4"] Sep 10 16:01:55.223592 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:55.223576 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-0e6e0dcc-86c4d5f9db-r7bz4" Sep 10 16:01:55.226985 ip-10-0-139-236 kubenswrapper[2576]: E0910 16:01:55.226954 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"payload-processing-e2e-mt-0e6e0dcc-dockercfg-n9wsr\" is forbidden: User \"system:node:ip-10-0-139-236.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-139-236.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-0e6e0dcc-dockercfg-n9wsr\"" type="*v1.Secret" Sep 10 16:01:55.226985 ip-10-0-139-236 kubenswrapper[2576]: E0910 16:01:55.226960 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"payload-processing-plugins-e2e-mt-0e6e0dcc\" is forbidden: User \"system:node:ip-10-0-139-236.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-139-236.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-0e6e0dcc\"" type="*v1.ConfigMap" Sep 10 16:01:55.227131 ip-10-0-139-236 kubenswrapper[2576]: E0910 16:01:55.227039 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"payload-pre-processing-e2e-mt-0e6e0dcc-86c4d5f9db-r7bz4\" is forbidden: User \"system:node:ip-10-0-139-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-139-236.ec2.internal' and this object" podUID="b170d249-cf04-458b-9d62-2030e939fae3" pod="openshift-ingress/payload-pre-processing-e2e-mt-0e6e0dcc-86c4d5f9db-r7bz4" Sep 10 16:01:55.247281 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:55.247254 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b170d249-cf04-458b-9d62-2030e939fae3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-0e6e0dcc-86c4d5f9db-r7bz4\" (UID: \"b170d249-cf04-458b-9d62-2030e939fae3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0e6e0dcc-86c4d5f9db-r7bz4" Sep 10 16:01:55.247387 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:55.247348 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8vl7\" (UniqueName: \"kubernetes.io/projected/b170d249-cf04-458b-9d62-2030e939fae3-kube-api-access-f8vl7\") pod \"payload-pre-processing-e2e-mt-0e6e0dcc-86c4d5f9db-r7bz4\" (UID: \"b170d249-cf04-458b-9d62-2030e939fae3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0e6e0dcc-86c4d5f9db-r7bz4" Sep 10 16:01:55.249979 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:55.249957 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0e6e0dcc-86c4d5f9db-r7bz4"] Sep 10 16:01:55.350369 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:55.350275 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b170d249-cf04-458b-9d62-2030e939fae3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-0e6e0dcc-86c4d5f9db-r7bz4\" (UID: \"b170d249-cf04-458b-9d62-2030e939fae3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0e6e0dcc-86c4d5f9db-r7bz4" Sep 10 16:01:55.350369 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:55.350323 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8vl7\" (UniqueName: \"kubernetes.io/projected/b170d249-cf04-458b-9d62-2030e939fae3-kube-api-access-f8vl7\") pod \"payload-pre-processing-e2e-mt-0e6e0dcc-86c4d5f9db-r7bz4\" (UID: \"b170d249-cf04-458b-9d62-2030e939fae3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0e6e0dcc-86c4d5f9db-r7bz4" Sep 10 16:01:55.362230 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:55.362198 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8vl7\" (UniqueName: \"kubernetes.io/projected/b170d249-cf04-458b-9d62-2030e939fae3-kube-api-access-f8vl7\") pod \"payload-pre-processing-e2e-mt-0e6e0dcc-86c4d5f9db-r7bz4\" (UID: \"b170d249-cf04-458b-9d62-2030e939fae3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0e6e0dcc-86c4d5f9db-r7bz4" Sep 10 16:01:55.990335 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:55.990294 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 16:01:56.294650 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:56.294622 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-0e6e0dcc-dockercfg-n9wsr\"" Sep 10 16:01:56.350934 ip-10-0-139-236 kubenswrapper[2576]: E0910 16:01:56.350902 2576 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-mt-0e6e0dcc: failed to sync configmap cache: timed out waiting for the condition Sep 10 16:01:56.351097 ip-10-0-139-236 kubenswrapper[2576]: E0910 16:01:56.350993 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b170d249-cf04-458b-9d62-2030e939fae3-plugins-config-volume podName:b170d249-cf04-458b-9d62-2030e939fae3 nodeName:}" failed. No retries permitted until 2026-09-10 16:01:56.850975467 +0000 UTC m=+1194.204494607 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/b170d249-cf04-458b-9d62-2030e939fae3-plugins-config-volume") pod "payload-pre-processing-e2e-mt-0e6e0dcc-86c4d5f9db-r7bz4" (UID: "b170d249-cf04-458b-9d62-2030e939fae3") : failed to sync configmap cache: timed out waiting for the condition Sep 10 16:01:56.641467 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:56.641397 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-0e6e0dcc\"" Sep 10 16:01:56.863517 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:56.863474 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b170d249-cf04-458b-9d62-2030e939fae3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-0e6e0dcc-86c4d5f9db-r7bz4\" (UID: \"b170d249-cf04-458b-9d62-2030e939fae3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0e6e0dcc-86c4d5f9db-r7bz4" Sep 10 16:01:56.864114 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:56.864096 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b170d249-cf04-458b-9d62-2030e939fae3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-0e6e0dcc-86c4d5f9db-r7bz4\" (UID: \"b170d249-cf04-458b-9d62-2030e939fae3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0e6e0dcc-86c4d5f9db-r7bz4" Sep 10 16:01:57.034596 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:57.034505 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-0e6e0dcc-86c4d5f9db-r7bz4" Sep 10 16:01:57.303551 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:57.303525 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0e6e0dcc-86c4d5f9db-r7bz4"] Sep 10 16:01:57.305466 ip-10-0-139-236 kubenswrapper[2576]: W0910 16:01:57.305438 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb170d249_cf04_458b_9d62_2030e939fae3.slice/crio-84d1e69cd7a78a3926dddb9276c64000af91ea4617a3ae366c07309b8f664de6 WatchSource:0}: Error finding container 84d1e69cd7a78a3926dddb9276c64000af91ea4617a3ae366c07309b8f664de6: Status 404 returned error can't find the container with id 84d1e69cd7a78a3926dddb9276c64000af91ea4617a3ae366c07309b8f664de6 Sep 10 16:01:58.284210 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:58.284175 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-0e6e0dcc-86c4d5f9db-r7bz4" event={"ID":"b170d249-cf04-458b-9d62-2030e939fae3","Type":"ContainerStarted","Data":"60d8169e6e94ab0d5ea4d5b1b8134e206fa04a87d3322499eb20c723bf3c0732"} Sep 10 16:01:58.284584 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:58.284239 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-0e6e0dcc-86c4d5f9db-r7bz4" Sep 10 16:01:58.284584 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:58.284261 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-0e6e0dcc-86c4d5f9db-r7bz4" event={"ID":"b170d249-cf04-458b-9d62-2030e939fae3","Type":"ContainerStarted","Data":"84d1e69cd7a78a3926dddb9276c64000af91ea4617a3ae366c07309b8f664de6"} Sep 10 16:01:58.304183 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:58.304132 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-0e6e0dcc-86c4d5f9db-r7bz4" podStartSLOduration=3.304118783 podStartE2EDuration="3.304118783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:01:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:01:58.3005217 +0000 UTC m=+1195.654040873" watchObservedRunningTime="2026-09-10 16:01:58.304118783 +0000 UTC m=+1195.657637944" Sep 10 16:01:59.326401 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:59.326357 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:01:59.395442 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:01:59.395403 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:09.291429 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:02:09.291389 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-0e6e0dcc-86c4d5f9db-r7bz4" Sep 10 16:02:15.289421 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:02:15.289387 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 16:02:18.297518 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:02:18.297471 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:33.077002 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:02:33.076970 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 16:02:34.671878 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:02:34.671839 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:52.692789 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:02:52.692750 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 16:03:03.314945 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:03:03.314859 2576 scope.go:117] "RemoveContainer" containerID="273587ce5455ed2b0a3ec21238d2e1c13e2c8b6769d381dd4613cf64cc3d689b" Sep 10 16:03:03.325542 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:03:03.325520 2576 scope.go:117] "RemoveContainer" containerID="58db7d05c5a033825a8ceb7369e5344fa79737cc9f28b2936870b7548c30a855" Sep 10 16:03:11.179942 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:03:11.179906 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 16:03:18.253481 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:03:18.253445 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0e6e0dcc-kntr2"] Sep 10 16:03:18.258637 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:03:18.258618 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0e6e0dcc-kntr2" Sep 10 16:03:18.261688 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:03:18.261668 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwwht\"" Sep 10 16:03:18.262086 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:03:18.262069 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 16:03:18.267606 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:03:18.267582 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0e6e0dcc-kntr2"] Sep 10 16:03:18.413129 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:03:18.413091 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6829132-3a31-40df-a00f-7375369477fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-0e6e0dcc-kntr2\" (UID: \"c6829132-3a31-40df-a00f-7375369477fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0e6e0dcc-kntr2" Sep 10 16:03:18.514569 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:03:18.514483 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6829132-3a31-40df-a00f-7375369477fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-0e6e0dcc-kntr2\" (UID: \"c6829132-3a31-40df-a00f-7375369477fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0e6e0dcc-kntr2" Sep 10 16:03:18.515121 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:03:18.515101 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6829132-3a31-40df-a00f-7375369477fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-0e6e0dcc-kntr2\" (UID: \"c6829132-3a31-40df-a00f-7375369477fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0e6e0dcc-kntr2" Sep 10 16:03:18.569782 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:03:18.569720 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0e6e0dcc-kntr2" Sep 10 16:03:18.847053 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:03:18.847028 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0e6e0dcc-kntr2"] Sep 10 16:03:18.849107 ip-10-0-139-236 kubenswrapper[2576]: W0910 16:03:18.849080 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6829132_3a31_40df_a00f_7375369477fd.slice/crio-67e889110a997db89fc42098978d72d81b86181b08adcede1b1f47c6d0396987 WatchSource:0}: Error finding container 67e889110a997db89fc42098978d72d81b86181b08adcede1b1f47c6d0396987: Status 404 returned error can't find the container with id 67e889110a997db89fc42098978d72d81b86181b08adcede1b1f47c6d0396987 Sep 10 16:03:19.737544 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:03:19.737512 2576 generic.go:356] "Generic (PLEG): container finished" podID="c6829132-3a31-40df-a00f-7375369477fd" containerID="59416fe0c08ffe67d661e47af765205d04e19a2c9467c2afa3ebbc101ac08ee8" exitCode=0 Sep 10 16:03:19.737914 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:03:19.737565 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0e6e0dcc-kntr2" event={"ID":"c6829132-3a31-40df-a00f-7375369477fd","Type":"ContainerDied","Data":"59416fe0c08ffe67d661e47af765205d04e19a2c9467c2afa3ebbc101ac08ee8"} Sep 10 16:03:19.737914 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:03:19.737604 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0e6e0dcc-kntr2" event={"ID":"c6829132-3a31-40df-a00f-7375369477fd","Type":"ContainerStarted","Data":"67e889110a997db89fc42098978d72d81b86181b08adcede1b1f47c6d0396987"} Sep 10 16:03:20.881792 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:03:20.881767 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0e6e0dcc-kntr2" Sep 10 16:03:20.937889 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:03:20.937854 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6829132-3a31-40df-a00f-7375369477fd-maas-api-service-ca\") pod \"c6829132-3a31-40df-a00f-7375369477fd\" (UID: \"c6829132-3a31-40df-a00f-7375369477fd\") " Sep 10 16:03:20.938303 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:03:20.938271 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6829132-3a31-40df-a00f-7375369477fd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c6829132-3a31-40df-a00f-7375369477fd" (UID: "c6829132-3a31-40df-a00f-7375369477fd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:03:21.039269 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:03:21.039240 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6829132-3a31-40df-a00f-7375369477fd-maas-api-service-ca\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:03:21.750962 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:03:21.750931 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0e6e0dcc-kntr2" Sep 10 16:03:21.751135 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:03:21.750933 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0e6e0dcc-kntr2" event={"ID":"c6829132-3a31-40df-a00f-7375369477fd","Type":"ContainerDied","Data":"67e889110a997db89fc42098978d72d81b86181b08adcede1b1f47c6d0396987"} Sep 10 16:03:21.751135 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:03:21.751054 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67e889110a997db89fc42098978d72d81b86181b08adcede1b1f47c6d0396987" Sep 10 16:03:23.268009 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:03:23.267977 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:28.087518 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:03:28.087480 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 16:03:33.945265 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:03:33.945226 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:33.983528 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:03:33.983484 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0e6e0dcc-86c4d5f9db-r7bz4"] Sep 10 16:03:33.983776 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:03:33.983720 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-0e6e0dcc-86c4d5f9db-r7bz4" podUID="b170d249-cf04-458b-9d62-2030e939fae3" containerName="payload-pre-processing" containerID="cri-o://60d8169e6e94ab0d5ea4d5b1b8134e206fa04a87d3322499eb20c723bf3c0732" gracePeriod=30 Sep 10 16:03:34.045153 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:03:34.045116 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:34.144694 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:03:34.144655 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:34.208617 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:03:34.208536 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0e6e0dcc-kntr2"] Sep 10 16:03:34.211823 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:03:34.211800 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0e6e0dcc-kntr2"] Sep 10 16:03:34.226631 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:03:34.226601 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:34.258509 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:03:34.258472 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-0e6e0dcc-86c4d5f9db-r7bz4" Sep 10 16:03:34.362178 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:03:34.362135 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f8vl7\" (UniqueName: \"kubernetes.io/projected/b170d249-cf04-458b-9d62-2030e939fae3-kube-api-access-f8vl7\") pod \"b170d249-cf04-458b-9d62-2030e939fae3\" (UID: \"b170d249-cf04-458b-9d62-2030e939fae3\") " Sep 10 16:03:34.362178 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:03:34.362189 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b170d249-cf04-458b-9d62-2030e939fae3-plugins-config-volume\") pod \"b170d249-cf04-458b-9d62-2030e939fae3\" (UID: \"b170d249-cf04-458b-9d62-2030e939fae3\") " Sep 10 16:03:34.362589 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:03:34.362564 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b170d249-cf04-458b-9d62-2030e939fae3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b170d249-cf04-458b-9d62-2030e939fae3" (UID: "b170d249-cf04-458b-9d62-2030e939fae3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:03:34.364572 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:03:34.364546 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b170d249-cf04-458b-9d62-2030e939fae3-kube-api-access-f8vl7" (OuterVolumeSpecName: "kube-api-access-f8vl7") pod "b170d249-cf04-458b-9d62-2030e939fae3" (UID: "b170d249-cf04-458b-9d62-2030e939fae3"). InnerVolumeSpecName "kube-api-access-f8vl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:03:34.463020 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:03:34.462930 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b170d249-cf04-458b-9d62-2030e939fae3-plugins-config-volume\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:03:34.463020 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:03:34.462958 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8vl7\" (UniqueName: \"kubernetes.io/projected/b170d249-cf04-458b-9d62-2030e939fae3-kube-api-access-f8vl7\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:03:34.828425 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:03:34.828387 2576 generic.go:356] "Generic (PLEG): container finished" podID="b170d249-cf04-458b-9d62-2030e939fae3" containerID="60d8169e6e94ab0d5ea4d5b1b8134e206fa04a87d3322499eb20c723bf3c0732" exitCode=0 Sep 10 16:03:34.828607 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:03:34.828437 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-0e6e0dcc-86c4d5f9db-r7bz4" Sep 10 16:03:34.828607 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:03:34.828456 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-0e6e0dcc-86c4d5f9db-r7bz4" event={"ID":"b170d249-cf04-458b-9d62-2030e939fae3","Type":"ContainerDied","Data":"60d8169e6e94ab0d5ea4d5b1b8134e206fa04a87d3322499eb20c723bf3c0732"} Sep 10 16:03:34.828607 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:03:34.828502 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-0e6e0dcc-86c4d5f9db-r7bz4" event={"ID":"b170d249-cf04-458b-9d62-2030e939fae3","Type":"ContainerDied","Data":"84d1e69cd7a78a3926dddb9276c64000af91ea4617a3ae366c07309b8f664de6"} Sep 10 16:03:34.828607 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:03:34.828524 2576 scope.go:117] "RemoveContainer" containerID="60d8169e6e94ab0d5ea4d5b1b8134e206fa04a87d3322499eb20c723bf3c0732" Sep 10 16:03:34.837996 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:03:34.837976 2576 scope.go:117] "RemoveContainer" containerID="60d8169e6e94ab0d5ea4d5b1b8134e206fa04a87d3322499eb20c723bf3c0732" Sep 10 16:03:34.838324 ip-10-0-139-236 kubenswrapper[2576]: E0910 16:03:34.838305 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60d8169e6e94ab0d5ea4d5b1b8134e206fa04a87d3322499eb20c723bf3c0732\": container with ID starting with 60d8169e6e94ab0d5ea4d5b1b8134e206fa04a87d3322499eb20c723bf3c0732 not found: ID does not exist" containerID="60d8169e6e94ab0d5ea4d5b1b8134e206fa04a87d3322499eb20c723bf3c0732" Sep 10 16:03:34.838391 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:03:34.838338 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60d8169e6e94ab0d5ea4d5b1b8134e206fa04a87d3322499eb20c723bf3c0732"} err="failed to get container status \"60d8169e6e94ab0d5ea4d5b1b8134e206fa04a87d3322499eb20c723bf3c0732\": rpc error: code = NotFound desc = could not find container \"60d8169e6e94ab0d5ea4d5b1b8134e206fa04a87d3322499eb20c723bf3c0732\": container with ID starting with 60d8169e6e94ab0d5ea4d5b1b8134e206fa04a87d3322499eb20c723bf3c0732 not found: ID does not exist" Sep 10 16:03:34.850329 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:03:34.850294 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0e6e0dcc-86c4d5f9db-r7bz4"] Sep 10 16:03:34.857785 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:03:34.857756 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0e6e0dcc-86c4d5f9db-r7bz4"] Sep 10 16:03:35.169284 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:03:35.169195 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b170d249-cf04-458b-9d62-2030e939fae3" path="/var/lib/kubelet/pods/b170d249-cf04-458b-9d62-2030e939fae3/volumes" Sep 10 16:03:35.169646 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:03:35.169554 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6829132-3a31-40df-a00f-7375369477fd" path="/var/lib/kubelet/pods/c6829132-3a31-40df-a00f-7375369477fd/volumes" Sep 10 16:03:37.441365 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:03:37.441319 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:42.603869 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:03:42.603827 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:45.904795 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:03:45.904767 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 16:03:57.278171 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:03:57.278135 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:58.214252 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:03:58.214209 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:04:02.382043 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:04:02.382008 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 16:04:03.348146 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:04:03.348118 2576 scope.go:117] "RemoveContainer" containerID="c609c6570d4f1dbf123ff666b74fc29583da5c6f5789b52073c1867d9f0700fd" Sep 10 16:04:09.719014 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:04:09.718972 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:04:15.985286 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:04:15.985248 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 16:04:18.730383 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:04:18.730346 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:04:20.060835 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:04:20.060793 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:04:26.911434 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:04:26.911347 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:04:49.100561 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:04:49.100523 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:04:51.719271 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:04:51.719235 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:04:53.925885 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:04:53.925848 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 16:04:55.732958 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:04:55.732924 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:04:55.922878 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:04:55.922843 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:05:02.465634 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:05:02.465583 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:05:02.564449 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:05:02.564404 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:05:03.140144 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:05:03.140109 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:05:03.372938 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:05:03.372899 2576 scope.go:117] "RemoveContainer" containerID="0a99e528629b4c0a3e8e5888df0cde3fd2b7ffb83e225f2376e273eb82537172" Sep 10 16:05:03.381978 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:05:03.381958 2576 scope.go:117] "RemoveContainer" containerID="238a3f729d7dbd156883f2f7fa8218eceb437d40fec279be77c3bbaed3055c83" Sep 10 16:05:09.856895 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:05:09.856864 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:05:09.995466 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:05:09.995401 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:05:10.128228 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:05:10.128134 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:05:19.202457 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:05:19.202419 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 16:05:20.662150 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:05:20.662116 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:05:20.916213 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:05:20.916104 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:05:23.712322 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:05:23.712291 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:05:35.489669 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:05:35.489636 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 16:05:41.810414 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:05:41.810374 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:05:45.493998 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:05:45.493966 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:05:46.367595 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:05:46.367557 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:05:48.296957 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:05:48.296926 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:05:48.449164 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:05:48.449123 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:05:48.673744 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:05:48.673643 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:05:48.893827 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:05:48.893795 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 16:05:49.026918 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:05:49.026824 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:05:49.440425 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:05:49.440390 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:05:51.970596 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:05:51.970562 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:05:52.095945 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:05:52.095911 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:05:53.177194 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:05:53.177161 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:05:53.268946 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:05:53.268903 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:05:53.438698 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:05:53.438591 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:05:53.844013 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:05:53.843977 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:05:53.944916 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:05:53.944873 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:05:53.980270 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:05:53.980235 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-45e544-6748d957fd-6dbcn"] Sep 10 16:05:53.980664 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:05:53.980652 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b170d249-cf04-458b-9d62-2030e939fae3" containerName="payload-pre-processing" Sep 10 16:05:53.980708 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:05:53.980666 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b170d249-cf04-458b-9d62-2030e939fae3" containerName="payload-pre-processing" Sep 10 16:05:53.980708 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:05:53.980685 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6829132-3a31-40df-a00f-7375369477fd" containerName="revoke-keys" Sep 10 16:05:53.980708 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:05:53.980690 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6829132-3a31-40df-a00f-7375369477fd" containerName="revoke-keys" Sep 10 16:05:53.980841 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:05:53.980763 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6829132-3a31-40df-a00f-7375369477fd" containerName="revoke-keys" Sep 10 16:05:53.980841 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:05:53.980775 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b170d249-cf04-458b-9d62-2030e939fae3" containerName="payload-pre-processing" Sep 10 16:05:53.984320 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:05:53.984297 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-45e544-6748d957fd-6dbcn" Sep 10 16:05:53.987339 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:05:53.987322 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-45e544\"" Sep 10 16:05:54.001576 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:05:54.001547 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-45e544-6748d957fd-6dbcn"] Sep 10 16:05:54.040900 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:05:54.040858 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c65f7930-58fe-4f93-8e68-6d7986b93927-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-45e544-6748d957fd-6dbcn\" (UID: \"c65f7930-58fe-4f93-8e68-6d7986b93927\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-45e544-6748d957fd-6dbcn" Sep 10 16:05:54.041076 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:05:54.040964 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncrds\" (UniqueName: \"kubernetes.io/projected/c65f7930-58fe-4f93-8e68-6d7986b93927-kube-api-access-ncrds\") pod \"payload-pre-processing-e2e-api-a-45e544-6748d957fd-6dbcn\" (UID: \"c65f7930-58fe-4f93-8e68-6d7986b93927\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-45e544-6748d957fd-6dbcn" Sep 10 16:05:54.142005 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:05:54.141917 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c65f7930-58fe-4f93-8e68-6d7986b93927-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-45e544-6748d957fd-6dbcn\" (UID: \"c65f7930-58fe-4f93-8e68-6d7986b93927\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-45e544-6748d957fd-6dbcn" Sep 10 16:05:54.142005 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:05:54.141985 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncrds\" (UniqueName: \"kubernetes.io/projected/c65f7930-58fe-4f93-8e68-6d7986b93927-kube-api-access-ncrds\") pod \"payload-pre-processing-e2e-api-a-45e544-6748d957fd-6dbcn\" (UID: \"c65f7930-58fe-4f93-8e68-6d7986b93927\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-45e544-6748d957fd-6dbcn" Sep 10 16:05:54.142719 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:05:54.142697 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c65f7930-58fe-4f93-8e68-6d7986b93927-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-45e544-6748d957fd-6dbcn\" (UID: \"c65f7930-58fe-4f93-8e68-6d7986b93927\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-45e544-6748d957fd-6dbcn" Sep 10 16:05:54.151673 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:05:54.151645 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncrds\" (UniqueName: \"kubernetes.io/projected/c65f7930-58fe-4f93-8e68-6d7986b93927-kube-api-access-ncrds\") pod \"payload-pre-processing-e2e-api-a-45e544-6748d957fd-6dbcn\" (UID: \"c65f7930-58fe-4f93-8e68-6d7986b93927\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-45e544-6748d957fd-6dbcn" Sep 10 16:05:54.294750 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:05:54.294658 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-45e544-6748d957fd-6dbcn" Sep 10 16:05:54.571670 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:05:54.571646 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-45e544-6748d957fd-6dbcn"] Sep 10 16:05:54.573821 ip-10-0-139-236 kubenswrapper[2576]: W0910 16:05:54.573792 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc65f7930_58fe_4f93_8e68_6d7986b93927.slice/crio-d3494531374f63f0e7c6ed01baf57b0c5103e3affba2f6902e98689c89ff9e7e WatchSource:0}: Error finding container d3494531374f63f0e7c6ed01baf57b0c5103e3affba2f6902e98689c89ff9e7e: Status 404 returned error can't find the container with id d3494531374f63f0e7c6ed01baf57b0c5103e3affba2f6902e98689c89ff9e7e Sep 10 16:05:54.583989 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:05:54.583965 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-45e544-6748d957fd-6dbcn" event={"ID":"c65f7930-58fe-4f93-8e68-6d7986b93927","Type":"ContainerStarted","Data":"d3494531374f63f0e7c6ed01baf57b0c5103e3affba2f6902e98689c89ff9e7e"} Sep 10 16:05:55.591121 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:05:55.591085 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-45e544-6748d957fd-6dbcn" event={"ID":"c65f7930-58fe-4f93-8e68-6d7986b93927","Type":"ContainerStarted","Data":"f6cebb21ef81af96ec29b547bb857e924e58b9e39d216626851820428d5db406"} Sep 10 16:05:55.591534 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:05:55.591166 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-45e544-6748d957fd-6dbcn" Sep 10 16:05:55.625407 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:05:55.625355 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-45e544-6748d957fd-6dbcn" podStartSLOduration=2.62534088 podStartE2EDuration="2.62534088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:05:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:05:55.621827318 +0000 UTC m=+1432.975346500" watchObservedRunningTime="2026-09-10 16:05:55.62534088 +0000 UTC m=+1432.978860041" Sep 10 16:06:03.404598 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:06:03.404564 2576 scope.go:117] "RemoveContainer" containerID="610db5d44c9a522e550c5130d92e152f4c7ba6ef1fb13ba90917e208cc83c474" Sep 10 16:06:06.598936 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:06:06.598905 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-45e544-6748d957fd-6dbcn" Sep 10 16:06:08.091436 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:06:08.091404 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 16:06:09.564715 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:06:09.564676 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:06:12.895184 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:06:12.895144 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:06:14.327770 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:06:14.327717 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:06:16.863408 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:06:16.863365 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:06:17.849362 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:06:17.849319 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:06:18.007257 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:06:18.007199 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:06:19.044559 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:06:19.044513 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:06:21.014434 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:06:21.014397 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:06:21.114476 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:06:21.114440 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:06:23.028870 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:06:23.028838 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:06:23.129043 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:06:23.129005 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-81d3ea-8649cf98c8-pr8lc"] Sep 10 16:06:23.132356 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:06:23.132340 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-81d3ea-8649cf98c8-pr8lc" Sep 10 16:06:23.134627 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:06:23.134607 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-81d3ea-dockercfg-wqc8b\"" Sep 10 16:06:23.134957 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:06:23.134940 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-81d3ea\"" Sep 10 16:06:23.147089 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:06:23.147066 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:06:23.157944 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:06:23.157918 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-81d3ea-8649cf98c8-pr8lc"] Sep 10 16:06:23.228067 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:06:23.228026 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/025fd18f-5774-45e3-b641-db5394c62e12-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-81d3ea-8649cf98c8-pr8lc\" (UID: \"025fd18f-5774-45e3-b641-db5394c62e12\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-81d3ea-8649cf98c8-pr8lc" Sep 10 16:06:23.231357 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:06:23.231323 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxbmh\" (UniqueName: \"kubernetes.io/projected/025fd18f-5774-45e3-b641-db5394c62e12-kube-api-access-sxbmh\") pod \"payload-pre-processing-e2e-api-b-81d3ea-8649cf98c8-pr8lc\" (UID: \"025fd18f-5774-45e3-b641-db5394c62e12\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-81d3ea-8649cf98c8-pr8lc" Sep 10 16:06:23.232352 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:06:23.232323 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-81d3ea-567fc5c4f7-tznbb"] Sep 10 16:06:23.237008 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:06:23.236984 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-81d3ea-567fc5c4f7-tznbb" Sep 10 16:06:23.258518 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:06:23.258077 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-81d3ea-567fc5c4f7-tznbb"] Sep 10 16:06:23.332749 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:06:23.332708 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c60d8e05-efb7-4e99-8b44-90aac55d3504-plugins-config-volume\") pod \"payload-processing-e2e-api-b-81d3ea-567fc5c4f7-tznbb\" (UID: \"c60d8e05-efb7-4e99-8b44-90aac55d3504\") " pod="openshift-ingress/payload-processing-e2e-api-b-81d3ea-567fc5c4f7-tznbb" Sep 10 16:06:23.332928 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:06:23.332780 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/025fd18f-5774-45e3-b641-db5394c62e12-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-81d3ea-8649cf98c8-pr8lc\" (UID: \"025fd18f-5774-45e3-b641-db5394c62e12\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-81d3ea-8649cf98c8-pr8lc" Sep 10 16:06:23.332928 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:06:23.332857 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxbmh\" (UniqueName: \"kubernetes.io/projected/025fd18f-5774-45e3-b641-db5394c62e12-kube-api-access-sxbmh\") pod \"payload-pre-processing-e2e-api-b-81d3ea-8649cf98c8-pr8lc\" (UID: \"025fd18f-5774-45e3-b641-db5394c62e12\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-81d3ea-8649cf98c8-pr8lc" Sep 10 16:06:23.332928 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:06:23.332873 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c98k7\" (UniqueName: \"kubernetes.io/projected/c60d8e05-efb7-4e99-8b44-90aac55d3504-kube-api-access-c98k7\") pod \"payload-processing-e2e-api-b-81d3ea-567fc5c4f7-tznbb\" (UID: \"c60d8e05-efb7-4e99-8b44-90aac55d3504\") " pod="openshift-ingress/payload-processing-e2e-api-b-81d3ea-567fc5c4f7-tznbb" Sep 10 16:06:23.333539 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:06:23.333521 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/025fd18f-5774-45e3-b641-db5394c62e12-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-81d3ea-8649cf98c8-pr8lc\" (UID: \"025fd18f-5774-45e3-b641-db5394c62e12\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-81d3ea-8649cf98c8-pr8lc" Sep 10 16:06:23.345285 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:06:23.345263 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxbmh\" (UniqueName: \"kubernetes.io/projected/025fd18f-5774-45e3-b641-db5394c62e12-kube-api-access-sxbmh\") pod \"payload-pre-processing-e2e-api-b-81d3ea-8649cf98c8-pr8lc\" (UID: \"025fd18f-5774-45e3-b641-db5394c62e12\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-81d3ea-8649cf98c8-pr8lc" Sep 10 16:06:23.434094 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:06:23.434059 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c60d8e05-efb7-4e99-8b44-90aac55d3504-plugins-config-volume\") pod \"payload-processing-e2e-api-b-81d3ea-567fc5c4f7-tznbb\" (UID: \"c60d8e05-efb7-4e99-8b44-90aac55d3504\") " pod="openshift-ingress/payload-processing-e2e-api-b-81d3ea-567fc5c4f7-tznbb" Sep 10 16:06:23.434259 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:06:23.434137 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c98k7\" (UniqueName: \"kubernetes.io/projected/c60d8e05-efb7-4e99-8b44-90aac55d3504-kube-api-access-c98k7\") pod \"payload-processing-e2e-api-b-81d3ea-567fc5c4f7-tznbb\" (UID: \"c60d8e05-efb7-4e99-8b44-90aac55d3504\") " pod="openshift-ingress/payload-processing-e2e-api-b-81d3ea-567fc5c4f7-tznbb" Sep 10 16:06:23.434850 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:06:23.434830 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c60d8e05-efb7-4e99-8b44-90aac55d3504-plugins-config-volume\") pod \"payload-processing-e2e-api-b-81d3ea-567fc5c4f7-tznbb\" (UID: \"c60d8e05-efb7-4e99-8b44-90aac55d3504\") " pod="openshift-ingress/payload-processing-e2e-api-b-81d3ea-567fc5c4f7-tznbb" Sep 10 16:06:23.446689 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:06:23.445226 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c98k7\" (UniqueName: \"kubernetes.io/projected/c60d8e05-efb7-4e99-8b44-90aac55d3504-kube-api-access-c98k7\") pod \"payload-processing-e2e-api-b-81d3ea-567fc5c4f7-tznbb\" (UID: \"c60d8e05-efb7-4e99-8b44-90aac55d3504\") " pod="openshift-ingress/payload-processing-e2e-api-b-81d3ea-567fc5c4f7-tznbb" Sep 10 16:06:23.446689 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:06:23.445416 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-81d3ea-8649cf98c8-pr8lc" Sep 10 16:06:23.552077 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:06:23.552042 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-81d3ea-567fc5c4f7-tznbb" Sep 10 16:06:23.733768 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:06:23.733721 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-81d3ea-8649cf98c8-pr8lc"] Sep 10 16:06:23.735339 ip-10-0-139-236 kubenswrapper[2576]: W0910 16:06:23.735312 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod025fd18f_5774_45e3_b641_db5394c62e12.slice/crio-e6c2ab969a1093a2fd6cedc3d6b258cf69259661cb0faa01001325da193004bc WatchSource:0}: Error finding container e6c2ab969a1093a2fd6cedc3d6b258cf69259661cb0faa01001325da193004bc: Status 404 returned error can't find the container with id e6c2ab969a1093a2fd6cedc3d6b258cf69259661cb0faa01001325da193004bc Sep 10 16:06:23.829606 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:06:23.829575 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-81d3ea-567fc5c4f7-tznbb"] Sep 10 16:06:23.834127 ip-10-0-139-236 kubenswrapper[2576]: W0910 16:06:23.834099 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc60d8e05_efb7_4e99_8b44_90aac55d3504.slice/crio-29fd6adff167d558e4662ce182cb1dbc47e18da25780f1af7377deed2b82361b WatchSource:0}: Error finding container 29fd6adff167d558e4662ce182cb1dbc47e18da25780f1af7377deed2b82361b: Status 404 returned error can't find the container with id 29fd6adff167d558e4662ce182cb1dbc47e18da25780f1af7377deed2b82361b Sep 10 16:06:24.747151 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:06:24.747115 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-81d3ea-8649cf98c8-pr8lc" event={"ID":"025fd18f-5774-45e3-b641-db5394c62e12","Type":"ContainerStarted","Data":"0995a15ef69174fd2d370c9eac796189dcda96cec4ab0cb87f31341342b88edf"} Sep 10 16:06:24.747630 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:06:24.747175 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-81d3ea-8649cf98c8-pr8lc" event={"ID":"025fd18f-5774-45e3-b641-db5394c62e12","Type":"ContainerStarted","Data":"e6c2ab969a1093a2fd6cedc3d6b258cf69259661cb0faa01001325da193004bc"} Sep 10 16:06:24.747630 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:06:24.747208 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-81d3ea-8649cf98c8-pr8lc" Sep 10 16:06:24.748791 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:06:24.748762 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-81d3ea-567fc5c4f7-tznbb" event={"ID":"c60d8e05-efb7-4e99-8b44-90aac55d3504","Type":"ContainerStarted","Data":"e665ae61a8d67671f6ac96941fbc0534cf42691c6f4e9a80044fc13a5499e7e5"} Sep 10 16:06:24.748903 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:06:24.748816 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-81d3ea-567fc5c4f7-tznbb" event={"ID":"c60d8e05-efb7-4e99-8b44-90aac55d3504","Type":"ContainerStarted","Data":"29fd6adff167d558e4662ce182cb1dbc47e18da25780f1af7377deed2b82361b"} Sep 10 16:06:24.748903 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:06:24.748844 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-81d3ea-567fc5c4f7-tznbb" Sep 10 16:06:24.774516 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:06:24.774463 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-81d3ea-8649cf98c8-pr8lc" podStartSLOduration=1.7744484520000001 podStartE2EDuration="1.774448452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:06:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:06:24.771269278 +0000 UTC m=+1462.124788455" watchObservedRunningTime="2026-09-10 16:06:24.774448452 +0000 UTC m=+1462.127967647" Sep 10 16:06:24.798337 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:06:24.798273 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-81d3ea-567fc5c4f7-tznbb" podStartSLOduration=1.798253253 podStartE2EDuration="1.798253253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:06:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:06:24.794424204 +0000 UTC m=+1462.147943402" watchObservedRunningTime="2026-09-10 16:06:24.798253253 +0000 UTC m=+1462.151772417" Sep 10 16:06:27.801012 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:06:27.800976 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 16:06:35.755934 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:06:35.755902 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-81d3ea-8649cf98c8-pr8lc" Sep 10 16:06:35.756327 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:06:35.755980 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-81d3ea-567fc5c4f7-tznbb" Sep 10 16:06:36.049494 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:06:36.049462 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:06:38.363210 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:06:38.363167 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:06:43.577899 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:06:43.577851 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:06:45.973637 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:06:45.973598 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:06:49.037159 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:06:49.037122 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:06:53.585006 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:06:53.584970 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 16:06:53.750550 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:06:53.750506 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:06:59.046172 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:06:59.046128 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:04.749319 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:04.749271 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:04.801434 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:04.801401 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-45e544-6748d957fd-6dbcn"] Sep 10 16:07:04.801719 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:04.801695 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-45e544-6748d957fd-6dbcn" podUID="c65f7930-58fe-4f93-8e68-6d7986b93927" containerName="payload-pre-processing" containerID="cri-o://f6cebb21ef81af96ec29b547bb857e924e58b9e39d216626851820428d5db406" gracePeriod=30 Sep 10 16:07:04.832650 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:04.832612 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:04.935189 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:04.935143 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:04.980814 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:04.980349 2576 generic.go:356] "Generic (PLEG): container finished" podID="c65f7930-58fe-4f93-8e68-6d7986b93927" containerID="f6cebb21ef81af96ec29b547bb857e924e58b9e39d216626851820428d5db406" exitCode=0 Sep 10 16:07:04.980814 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:04.980469 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-45e544-6748d957fd-6dbcn" event={"ID":"c65f7930-58fe-4f93-8e68-6d7986b93927","Type":"ContainerDied","Data":"f6cebb21ef81af96ec29b547bb857e924e58b9e39d216626851820428d5db406"} Sep 10 16:07:05.085863 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:05.085832 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-45e544-6748d957fd-6dbcn" Sep 10 16:07:05.156331 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:05.156297 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c65f7930-58fe-4f93-8e68-6d7986b93927-plugins-config-volume\") pod \"c65f7930-58fe-4f93-8e68-6d7986b93927\" (UID: \"c65f7930-58fe-4f93-8e68-6d7986b93927\") " Sep 10 16:07:05.156331 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:05.156340 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ncrds\" (UniqueName: \"kubernetes.io/projected/c65f7930-58fe-4f93-8e68-6d7986b93927-kube-api-access-ncrds\") pod \"c65f7930-58fe-4f93-8e68-6d7986b93927\" (UID: \"c65f7930-58fe-4f93-8e68-6d7986b93927\") " Sep 10 16:07:05.156826 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:05.156789 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c65f7930-58fe-4f93-8e68-6d7986b93927-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c65f7930-58fe-4f93-8e68-6d7986b93927" (UID: "c65f7930-58fe-4f93-8e68-6d7986b93927"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:07:05.159820 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:05.159785 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c65f7930-58fe-4f93-8e68-6d7986b93927-kube-api-access-ncrds" (OuterVolumeSpecName: "kube-api-access-ncrds") pod "c65f7930-58fe-4f93-8e68-6d7986b93927" (UID: "c65f7930-58fe-4f93-8e68-6d7986b93927"). InnerVolumeSpecName "kube-api-access-ncrds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:07:05.214832 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:05.214794 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:05.258432 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:05.258395 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c65f7930-58fe-4f93-8e68-6d7986b93927-plugins-config-volume\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:07:05.258432 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:05.258431 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncrds\" (UniqueName: \"kubernetes.io/projected/c65f7930-58fe-4f93-8e68-6d7986b93927-kube-api-access-ncrds\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:07:05.328069 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:05.327976 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:05.419874 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:05.419840 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:05.986887 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:05.986858 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-45e544-6748d957fd-6dbcn" Sep 10 16:07:05.987394 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:05.986889 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-45e544-6748d957fd-6dbcn" event={"ID":"c65f7930-58fe-4f93-8e68-6d7986b93927","Type":"ContainerDied","Data":"d3494531374f63f0e7c6ed01baf57b0c5103e3affba2f6902e98689c89ff9e7e"} Sep 10 16:07:05.987394 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:05.986956 2576 scope.go:117] "RemoveContainer" containerID="f6cebb21ef81af96ec29b547bb857e924e58b9e39d216626851820428d5db406" Sep 10 16:07:06.011639 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:06.011598 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-45e544-6748d957fd-6dbcn"] Sep 10 16:07:06.015648 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:06.015611 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-45e544-6748d957fd-6dbcn"] Sep 10 16:07:07.170854 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:07.170817 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c65f7930-58fe-4f93-8e68-6d7986b93927" path="/var/lib/kubelet/pods/c65f7930-58fe-4f93-8e68-6d7986b93927/volumes" Sep 10 16:07:10.922092 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:10.922057 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 16:07:11.590900 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:11.590858 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:11.691352 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:11.691302 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:11.864957 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:11.864850 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:12.402423 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:12.402390 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:15.248218 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:15.248162 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:17.369405 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:17.369354 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:18.377624 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:18.377588 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:18.815355 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:18.815316 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:19.237975 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:19.237845 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:19.650940 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:19.650902 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:22.633982 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:22.633942 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:22.729861 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:22.729828 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:23.649720 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:23.649680 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:23.745100 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:23.745066 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:23.844477 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:23.844439 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:24.183718 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:24.183676 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:24.281167 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:24.281064 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:24.321714 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:24.321675 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c0ec41-69664b596f-642w8"] Sep 10 16:07:24.322219 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:24.322203 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c65f7930-58fe-4f93-8e68-6d7986b93927" containerName="payload-pre-processing" Sep 10 16:07:24.322269 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:24.322221 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c65f7930-58fe-4f93-8e68-6d7986b93927" containerName="payload-pre-processing" Sep 10 16:07:24.322305 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:24.322290 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c65f7930-58fe-4f93-8e68-6d7986b93927" containerName="payload-pre-processing" Sep 10 16:07:24.326990 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:24.326970 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c0ec41-69664b596f-642w8" Sep 10 16:07:24.329939 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:24.329907 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-c0ec41\"" Sep 10 16:07:24.330087 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:24.329959 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-c0ec41-dockercfg-fnc9z\"" Sep 10 16:07:24.337051 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:24.337023 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c0ec41-69664b596f-642w8"] Sep 10 16:07:24.359772 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:24.359724 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8n5g\" (UniqueName: \"kubernetes.io/projected/168d3721-ce5b-42b7-b3c5-44106c77ac3f-kube-api-access-r8n5g\") pod \"payload-processing-e2e-ipp-a-c0ec41-69664b596f-642w8\" (UID: \"168d3721-ce5b-42b7-b3c5-44106c77ac3f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c0ec41-69664b596f-642w8" Sep 10 16:07:24.359951 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:24.359902 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/168d3721-ce5b-42b7-b3c5-44106c77ac3f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c0ec41-69664b596f-642w8\" (UID: \"168d3721-ce5b-42b7-b3c5-44106c77ac3f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c0ec41-69664b596f-642w8" Sep 10 16:07:24.461159 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:24.461061 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/168d3721-ce5b-42b7-b3c5-44106c77ac3f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c0ec41-69664b596f-642w8\" (UID: \"168d3721-ce5b-42b7-b3c5-44106c77ac3f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c0ec41-69664b596f-642w8" Sep 10 16:07:24.462537 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:24.462395 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/168d3721-ce5b-42b7-b3c5-44106c77ac3f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c0ec41-69664b596f-642w8\" (UID: \"168d3721-ce5b-42b7-b3c5-44106c77ac3f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c0ec41-69664b596f-642w8" Sep 10 16:07:24.462766 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:24.462610 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8n5g\" (UniqueName: \"kubernetes.io/projected/168d3721-ce5b-42b7-b3c5-44106c77ac3f-kube-api-access-r8n5g\") pod \"payload-processing-e2e-ipp-a-c0ec41-69664b596f-642w8\" (UID: \"168d3721-ce5b-42b7-b3c5-44106c77ac3f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c0ec41-69664b596f-642w8" Sep 10 16:07:24.475433 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:24.475403 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8n5g\" (UniqueName: \"kubernetes.io/projected/168d3721-ce5b-42b7-b3c5-44106c77ac3f-kube-api-access-r8n5g\") pod \"payload-processing-e2e-ipp-a-c0ec41-69664b596f-642w8\" (UID: \"168d3721-ce5b-42b7-b3c5-44106c77ac3f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c0ec41-69664b596f-642w8" Sep 10 16:07:24.639573 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:24.639528 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c0ec41-69664b596f-642w8" Sep 10 16:07:24.932130 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:24.932078 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c0ec41-69664b596f-642w8"] Sep 10 16:07:24.932506 ip-10-0-139-236 kubenswrapper[2576]: W0910 16:07:24.932304 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod168d3721_ce5b_42b7_b3c5_44106c77ac3f.slice/crio-67b45f2aa2869fbe81b5749b74af2816fdb74027b7b4e006f76332a02546f540 WatchSource:0}: Error finding container 67b45f2aa2869fbe81b5749b74af2816fdb74027b7b4e006f76332a02546f540: Status 404 returned error can't find the container with id 67b45f2aa2869fbe81b5749b74af2816fdb74027b7b4e006f76332a02546f540 Sep 10 16:07:25.095311 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:25.095272 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c0ec41-69664b596f-642w8" event={"ID":"168d3721-ce5b-42b7-b3c5-44106c77ac3f","Type":"ContainerStarted","Data":"a5df31be123029b05f86acd3e73374c2a1e2602d238aeb08247900f6e7277787"} Sep 10 16:07:25.095544 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:25.095343 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c0ec41-69664b596f-642w8" Sep 10 16:07:25.095544 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:25.095368 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c0ec41-69664b596f-642w8" event={"ID":"168d3721-ce5b-42b7-b3c5-44106c77ac3f","Type":"ContainerStarted","Data":"67b45f2aa2869fbe81b5749b74af2816fdb74027b7b4e006f76332a02546f540"} Sep 10 16:07:25.118020 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:25.117965 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-c0ec41-69664b596f-642w8" podStartSLOduration=1.117926604 podStartE2EDuration="1.117926604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:07:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:07:25.115774897 +0000 UTC m=+1522.469294071" watchObservedRunningTime="2026-09-10 16:07:25.117926604 +0000 UTC m=+1522.471445776" Sep 10 16:07:25.308190 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:25.308150 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 16:07:28.035474 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:28.035430 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:28.065277 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:28.065243 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-81d3ea-567fc5c4f7-tznbb"] Sep 10 16:07:28.065534 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:28.065503 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-81d3ea-567fc5c4f7-tznbb" podUID="c60d8e05-efb7-4e99-8b44-90aac55d3504" containerName="payload-processing" containerID="cri-o://e665ae61a8d67671f6ac96941fbc0534cf42691c6f4e9a80044fc13a5499e7e5" gracePeriod=30 Sep 10 16:07:28.075163 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:28.075129 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-81d3ea-8649cf98c8-pr8lc"] Sep 10 16:07:28.075443 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:28.075406 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-81d3ea-8649cf98c8-pr8lc" podUID="025fd18f-5774-45e3-b641-db5394c62e12" containerName="payload-pre-processing" containerID="cri-o://0995a15ef69174fd2d370c9eac796189dcda96cec4ab0cb87f31341342b88edf" gracePeriod=30 Sep 10 16:07:28.140646 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:28.140611 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:28.242520 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:28.242460 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:28.339461 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:28.339400 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:28.352117 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:28.352090 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-81d3ea-567fc5c4f7-tznbb" Sep 10 16:07:28.383478 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:28.383451 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-81d3ea-8649cf98c8-pr8lc" Sep 10 16:07:28.407091 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:28.407058 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c60d8e05-efb7-4e99-8b44-90aac55d3504-plugins-config-volume\") pod \"c60d8e05-efb7-4e99-8b44-90aac55d3504\" (UID: \"c60d8e05-efb7-4e99-8b44-90aac55d3504\") " Sep 10 16:07:28.407283 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:28.407167 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/025fd18f-5774-45e3-b641-db5394c62e12-plugins-config-volume\") pod \"025fd18f-5774-45e3-b641-db5394c62e12\" (UID: \"025fd18f-5774-45e3-b641-db5394c62e12\") " Sep 10 16:07:28.407283 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:28.407205 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c98k7\" (UniqueName: \"kubernetes.io/projected/c60d8e05-efb7-4e99-8b44-90aac55d3504-kube-api-access-c98k7\") pod \"c60d8e05-efb7-4e99-8b44-90aac55d3504\" (UID: \"c60d8e05-efb7-4e99-8b44-90aac55d3504\") " Sep 10 16:07:28.407283 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:28.407251 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxbmh\" (UniqueName: \"kubernetes.io/projected/025fd18f-5774-45e3-b641-db5394c62e12-kube-api-access-sxbmh\") pod \"025fd18f-5774-45e3-b641-db5394c62e12\" (UID: \"025fd18f-5774-45e3-b641-db5394c62e12\") " Sep 10 16:07:28.407678 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:28.407536 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c60d8e05-efb7-4e99-8b44-90aac55d3504-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c60d8e05-efb7-4e99-8b44-90aac55d3504" (UID: "c60d8e05-efb7-4e99-8b44-90aac55d3504"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:07:28.407678 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:28.407608 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c60d8e05-efb7-4e99-8b44-90aac55d3504-plugins-config-volume\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:07:28.407678 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:28.407634 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/025fd18f-5774-45e3-b641-db5394c62e12-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "025fd18f-5774-45e3-b641-db5394c62e12" (UID: "025fd18f-5774-45e3-b641-db5394c62e12"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:07:28.410489 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:28.410453 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c60d8e05-efb7-4e99-8b44-90aac55d3504-kube-api-access-c98k7" (OuterVolumeSpecName: "kube-api-access-c98k7") pod "c60d8e05-efb7-4e99-8b44-90aac55d3504" (UID: "c60d8e05-efb7-4e99-8b44-90aac55d3504"). InnerVolumeSpecName "kube-api-access-c98k7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:07:28.411207 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:28.411171 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/025fd18f-5774-45e3-b641-db5394c62e12-kube-api-access-sxbmh" (OuterVolumeSpecName: "kube-api-access-sxbmh") pod "025fd18f-5774-45e3-b641-db5394c62e12" (UID: "025fd18f-5774-45e3-b641-db5394c62e12"). InnerVolumeSpecName "kube-api-access-sxbmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:07:28.508495 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:28.508464 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/025fd18f-5774-45e3-b641-db5394c62e12-plugins-config-volume\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:07:28.508495 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:28.508493 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c98k7\" (UniqueName: \"kubernetes.io/projected/c60d8e05-efb7-4e99-8b44-90aac55d3504-kube-api-access-c98k7\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:07:28.508495 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:28.508502 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxbmh\" (UniqueName: \"kubernetes.io/projected/025fd18f-5774-45e3-b641-db5394c62e12-kube-api-access-sxbmh\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:07:28.651803 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:28.651690 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:29.120689 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:29.120648 2576 generic.go:356] "Generic (PLEG): container finished" podID="c60d8e05-efb7-4e99-8b44-90aac55d3504" containerID="e665ae61a8d67671f6ac96941fbc0534cf42691c6f4e9a80044fc13a5499e7e5" exitCode=0 Sep 10 16:07:29.121228 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:29.120698 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-81d3ea-567fc5c4f7-tznbb" Sep 10 16:07:29.121228 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:29.120767 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-81d3ea-567fc5c4f7-tznbb" event={"ID":"c60d8e05-efb7-4e99-8b44-90aac55d3504","Type":"ContainerDied","Data":"e665ae61a8d67671f6ac96941fbc0534cf42691c6f4e9a80044fc13a5499e7e5"} Sep 10 16:07:29.121228 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:29.120833 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-81d3ea-567fc5c4f7-tznbb" event={"ID":"c60d8e05-efb7-4e99-8b44-90aac55d3504","Type":"ContainerDied","Data":"29fd6adff167d558e4662ce182cb1dbc47e18da25780f1af7377deed2b82361b"} Sep 10 16:07:29.121228 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:29.120868 2576 scope.go:117] "RemoveContainer" containerID="e665ae61a8d67671f6ac96941fbc0534cf42691c6f4e9a80044fc13a5499e7e5" Sep 10 16:07:29.122470 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:29.122449 2576 generic.go:356] "Generic (PLEG): container finished" podID="025fd18f-5774-45e3-b641-db5394c62e12" containerID="0995a15ef69174fd2d370c9eac796189dcda96cec4ab0cb87f31341342b88edf" exitCode=0 Sep 10 16:07:29.122594 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:29.122499 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-81d3ea-8649cf98c8-pr8lc" Sep 10 16:07:29.122594 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:29.122502 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-81d3ea-8649cf98c8-pr8lc" event={"ID":"025fd18f-5774-45e3-b641-db5394c62e12","Type":"ContainerDied","Data":"0995a15ef69174fd2d370c9eac796189dcda96cec4ab0cb87f31341342b88edf"} Sep 10 16:07:29.122594 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:29.122544 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-81d3ea-8649cf98c8-pr8lc" event={"ID":"025fd18f-5774-45e3-b641-db5394c62e12","Type":"ContainerDied","Data":"e6c2ab969a1093a2fd6cedc3d6b258cf69259661cb0faa01001325da193004bc"} Sep 10 16:07:29.132002 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:29.131974 2576 scope.go:117] "RemoveContainer" containerID="e665ae61a8d67671f6ac96941fbc0534cf42691c6f4e9a80044fc13a5499e7e5" Sep 10 16:07:29.132465 ip-10-0-139-236 kubenswrapper[2576]: E0910 16:07:29.132316 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e665ae61a8d67671f6ac96941fbc0534cf42691c6f4e9a80044fc13a5499e7e5\": container with ID starting with e665ae61a8d67671f6ac96941fbc0534cf42691c6f4e9a80044fc13a5499e7e5 not found: ID does not exist" containerID="e665ae61a8d67671f6ac96941fbc0534cf42691c6f4e9a80044fc13a5499e7e5" Sep 10 16:07:29.132465 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:29.132351 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e665ae61a8d67671f6ac96941fbc0534cf42691c6f4e9a80044fc13a5499e7e5"} err="failed to get container status \"e665ae61a8d67671f6ac96941fbc0534cf42691c6f4e9a80044fc13a5499e7e5\": rpc error: code = NotFound desc = could not find container \"e665ae61a8d67671f6ac96941fbc0534cf42691c6f4e9a80044fc13a5499e7e5\": container with ID starting with e665ae61a8d67671f6ac96941fbc0534cf42691c6f4e9a80044fc13a5499e7e5 not found: ID does not exist" Sep 10 16:07:29.132465 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:29.132372 2576 scope.go:117] "RemoveContainer" containerID="0995a15ef69174fd2d370c9eac796189dcda96cec4ab0cb87f31341342b88edf" Sep 10 16:07:29.145809 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:29.145569 2576 scope.go:117] "RemoveContainer" containerID="0995a15ef69174fd2d370c9eac796189dcda96cec4ab0cb87f31341342b88edf" Sep 10 16:07:29.146092 ip-10-0-139-236 kubenswrapper[2576]: E0910 16:07:29.146064 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0995a15ef69174fd2d370c9eac796189dcda96cec4ab0cb87f31341342b88edf\": container with ID starting with 0995a15ef69174fd2d370c9eac796189dcda96cec4ab0cb87f31341342b88edf not found: ID does not exist" containerID="0995a15ef69174fd2d370c9eac796189dcda96cec4ab0cb87f31341342b88edf" Sep 10 16:07:29.146188 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:29.146106 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0995a15ef69174fd2d370c9eac796189dcda96cec4ab0cb87f31341342b88edf"} err="failed to get container status \"0995a15ef69174fd2d370c9eac796189dcda96cec4ab0cb87f31341342b88edf\": rpc error: code = NotFound desc = could not find container \"0995a15ef69174fd2d370c9eac796189dcda96cec4ab0cb87f31341342b88edf\": container with ID starting with 0995a15ef69174fd2d370c9eac796189dcda96cec4ab0cb87f31341342b88edf not found: ID does not exist" Sep 10 16:07:29.148559 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:29.148536 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-81d3ea-8649cf98c8-pr8lc"] Sep 10 16:07:29.152068 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:29.152040 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-81d3ea-8649cf98c8-pr8lc"] Sep 10 16:07:29.170217 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:29.170181 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="025fd18f-5774-45e3-b641-db5394c62e12" path="/var/lib/kubelet/pods/025fd18f-5774-45e3-b641-db5394c62e12/volumes" Sep 10 16:07:29.170475 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:29.170461 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-81d3ea-567fc5c4f7-tznbb"] Sep 10 16:07:29.170661 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:29.170641 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-81d3ea-567fc5c4f7-tznbb"] Sep 10 16:07:31.169314 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:31.169279 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c60d8e05-efb7-4e99-8b44-90aac55d3504" path="/var/lib/kubelet/pods/c60d8e05-efb7-4e99-8b44-90aac55d3504/volumes" Sep 10 16:07:35.197442 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:35.197403 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:35.319948 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:35.319903 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:35.521558 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:35.521451 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:36.018775 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:36.018719 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:36.102914 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:36.102880 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c0ec41-69664b596f-642w8" Sep 10 16:07:38.536859 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:38.536821 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:41.284808 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:41.284774 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 16:07:44.642110 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:44.642058 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:46.179803 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:46.179764 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:46.219646 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:46.219613 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-a361a9-9f6d5f9c5-gs8hx"] Sep 10 16:07:46.219918 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:46.219888 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a361a9-9f6d5f9c5-gs8hx" podUID="a79ea085-ab37-462b-aa56-81f09c2eb640" containerName="payload-processing" containerID="cri-o://a3e444c0fd5b49fb77d1856058b4f8ed24d8deaadddce92514245fd6758d9e18" gracePeriod=30 Sep 10 16:07:46.232038 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:46.232008 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a361a9-7f976dfdc8-7hbtq"] Sep 10 16:07:46.232282 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:46.232252 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a361a9-7f976dfdc8-7hbtq" podUID="0f4b64a3-888f-42b9-be57-c38f5c500873" containerName="payload-pre-processing" containerID="cri-o://ca906a7e2619c66ec5dae61e950ddc40cb66ba44a9248ea540581a02cfca80bf" gracePeriod=30 Sep 10 16:07:46.266993 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:46.266956 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:46.344440 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:46.344406 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:46.433901 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:46.433811 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:46.528987 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:46.528950 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a361a9-9f6d5f9c5-gs8hx" Sep 10 16:07:46.531948 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:46.531915 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a361a9-7f976dfdc8-7hbtq" Sep 10 16:07:46.701784 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:46.701666 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a79ea085-ab37-462b-aa56-81f09c2eb640-plugins-config-volume\") pod \"a79ea085-ab37-462b-aa56-81f09c2eb640\" (UID: \"a79ea085-ab37-462b-aa56-81f09c2eb640\") " Sep 10 16:07:46.701784 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:46.701708 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f4b64a3-888f-42b9-be57-c38f5c500873-plugins-config-volume\") pod \"0f4b64a3-888f-42b9-be57-c38f5c500873\" (UID: \"0f4b64a3-888f-42b9-be57-c38f5c500873\") " Sep 10 16:07:46.701784 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:46.701780 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7zcgn\" (UniqueName: \"kubernetes.io/projected/0f4b64a3-888f-42b9-be57-c38f5c500873-kube-api-access-7zcgn\") pod \"0f4b64a3-888f-42b9-be57-c38f5c500873\" (UID: \"0f4b64a3-888f-42b9-be57-c38f5c500873\") " Sep 10 16:07:46.702082 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:46.701811 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kjjmk\" (UniqueName: \"kubernetes.io/projected/a79ea085-ab37-462b-aa56-81f09c2eb640-kube-api-access-kjjmk\") pod \"a79ea085-ab37-462b-aa56-81f09c2eb640\" (UID: \"a79ea085-ab37-462b-aa56-81f09c2eb640\") " Sep 10 16:07:46.702082 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:46.702025 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a79ea085-ab37-462b-aa56-81f09c2eb640-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a79ea085-ab37-462b-aa56-81f09c2eb640" (UID: "a79ea085-ab37-462b-aa56-81f09c2eb640"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:07:46.702193 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:46.702160 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a79ea085-ab37-462b-aa56-81f09c2eb640-plugins-config-volume\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:07:46.702193 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:46.702160 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f4b64a3-888f-42b9-be57-c38f5c500873-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0f4b64a3-888f-42b9-be57-c38f5c500873" (UID: "0f4b64a3-888f-42b9-be57-c38f5c500873"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:07:46.704213 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:46.704194 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a79ea085-ab37-462b-aa56-81f09c2eb640-kube-api-access-kjjmk" (OuterVolumeSpecName: "kube-api-access-kjjmk") pod "a79ea085-ab37-462b-aa56-81f09c2eb640" (UID: "a79ea085-ab37-462b-aa56-81f09c2eb640"). InnerVolumeSpecName "kube-api-access-kjjmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:07:46.704291 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:46.704215 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f4b64a3-888f-42b9-be57-c38f5c500873-kube-api-access-7zcgn" (OuterVolumeSpecName: "kube-api-access-7zcgn") pod "0f4b64a3-888f-42b9-be57-c38f5c500873" (UID: "0f4b64a3-888f-42b9-be57-c38f5c500873"). InnerVolumeSpecName "kube-api-access-7zcgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:07:46.803474 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:46.803434 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjjmk\" (UniqueName: \"kubernetes.io/projected/a79ea085-ab37-462b-aa56-81f09c2eb640-kube-api-access-kjjmk\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:07:46.803474 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:46.803467 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f4b64a3-888f-42b9-be57-c38f5c500873-plugins-config-volume\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:07:46.803474 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:46.803476 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zcgn\" (UniqueName: \"kubernetes.io/projected/0f4b64a3-888f-42b9-be57-c38f5c500873-kube-api-access-7zcgn\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:07:46.841786 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:46.841747 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:47.231588 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:47.231547 2576 generic.go:356] "Generic (PLEG): container finished" podID="a79ea085-ab37-462b-aa56-81f09c2eb640" containerID="a3e444c0fd5b49fb77d1856058b4f8ed24d8deaadddce92514245fd6758d9e18" exitCode=0 Sep 10 16:07:47.232060 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:47.231612 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a361a9-9f6d5f9c5-gs8hx" Sep 10 16:07:47.232060 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:47.231631 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a361a9-9f6d5f9c5-gs8hx" event={"ID":"a79ea085-ab37-462b-aa56-81f09c2eb640","Type":"ContainerDied","Data":"a3e444c0fd5b49fb77d1856058b4f8ed24d8deaadddce92514245fd6758d9e18"} Sep 10 16:07:47.232060 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:47.231688 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a361a9-9f6d5f9c5-gs8hx" event={"ID":"a79ea085-ab37-462b-aa56-81f09c2eb640","Type":"ContainerDied","Data":"57ab541628409da6ae198d651c55c5e2f85eef583aec5952ff1ee0214d90c10d"} Sep 10 16:07:47.232060 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:47.231717 2576 scope.go:117] "RemoveContainer" containerID="a3e444c0fd5b49fb77d1856058b4f8ed24d8deaadddce92514245fd6758d9e18" Sep 10 16:07:47.233351 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:47.233327 2576 generic.go:356] "Generic (PLEG): container finished" podID="0f4b64a3-888f-42b9-be57-c38f5c500873" containerID="ca906a7e2619c66ec5dae61e950ddc40cb66ba44a9248ea540581a02cfca80bf" exitCode=0 Sep 10 16:07:47.233494 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:47.233359 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a361a9-7f976dfdc8-7hbtq" event={"ID":"0f4b64a3-888f-42b9-be57-c38f5c500873","Type":"ContainerDied","Data":"ca906a7e2619c66ec5dae61e950ddc40cb66ba44a9248ea540581a02cfca80bf"} Sep 10 16:07:47.233494 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:47.233375 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a361a9-7f976dfdc8-7hbtq" Sep 10 16:07:47.233494 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:47.233459 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a361a9-7f976dfdc8-7hbtq" event={"ID":"0f4b64a3-888f-42b9-be57-c38f5c500873","Type":"ContainerDied","Data":"16263cce5f8626be71b70c8a5c1278160a6f9b2ca4b839215e33932fbec0eeb1"} Sep 10 16:07:47.245632 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:47.245612 2576 scope.go:117] "RemoveContainer" containerID="a3e444c0fd5b49fb77d1856058b4f8ed24d8deaadddce92514245fd6758d9e18" Sep 10 16:07:47.245991 ip-10-0-139-236 kubenswrapper[2576]: E0910 16:07:47.245971 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3e444c0fd5b49fb77d1856058b4f8ed24d8deaadddce92514245fd6758d9e18\": container with ID starting with a3e444c0fd5b49fb77d1856058b4f8ed24d8deaadddce92514245fd6758d9e18 not found: ID does not exist" containerID="a3e444c0fd5b49fb77d1856058b4f8ed24d8deaadddce92514245fd6758d9e18" Sep 10 16:07:47.246102 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:47.246004 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3e444c0fd5b49fb77d1856058b4f8ed24d8deaadddce92514245fd6758d9e18"} err="failed to get container status \"a3e444c0fd5b49fb77d1856058b4f8ed24d8deaadddce92514245fd6758d9e18\": rpc error: code = NotFound desc = could not find container \"a3e444c0fd5b49fb77d1856058b4f8ed24d8deaadddce92514245fd6758d9e18\": container with ID starting with a3e444c0fd5b49fb77d1856058b4f8ed24d8deaadddce92514245fd6758d9e18 not found: ID does not exist" Sep 10 16:07:47.246102 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:47.246029 2576 scope.go:117] "RemoveContainer" containerID="ca906a7e2619c66ec5dae61e950ddc40cb66ba44a9248ea540581a02cfca80bf" Sep 10 16:07:47.255116 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:47.255091 2576 scope.go:117] "RemoveContainer" containerID="ca906a7e2619c66ec5dae61e950ddc40cb66ba44a9248ea540581a02cfca80bf" Sep 10 16:07:47.255432 ip-10-0-139-236 kubenswrapper[2576]: E0910 16:07:47.255412 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca906a7e2619c66ec5dae61e950ddc40cb66ba44a9248ea540581a02cfca80bf\": container with ID starting with ca906a7e2619c66ec5dae61e950ddc40cb66ba44a9248ea540581a02cfca80bf not found: ID does not exist" containerID="ca906a7e2619c66ec5dae61e950ddc40cb66ba44a9248ea540581a02cfca80bf" Sep 10 16:07:47.255497 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:47.255441 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca906a7e2619c66ec5dae61e950ddc40cb66ba44a9248ea540581a02cfca80bf"} err="failed to get container status \"ca906a7e2619c66ec5dae61e950ddc40cb66ba44a9248ea540581a02cfca80bf\": rpc error: code = NotFound desc = could not find container \"ca906a7e2619c66ec5dae61e950ddc40cb66ba44a9248ea540581a02cfca80bf\": container with ID starting with ca906a7e2619c66ec5dae61e950ddc40cb66ba44a9248ea540581a02cfca80bf not found: ID does not exist" Sep 10 16:07:47.273759 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:47.273712 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-a361a9-9f6d5f9c5-gs8hx"] Sep 10 16:07:47.276910 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:47.276874 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-a361a9-9f6d5f9c5-gs8hx"] Sep 10 16:07:47.302096 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:47.302069 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a361a9-7f976dfdc8-7hbtq"] Sep 10 16:07:47.308814 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:47.308786 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a361a9-7f976dfdc8-7hbtq"] Sep 10 16:07:47.348553 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:47.348512 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:47.955905 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:47.955870 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:48.235085 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:48.235006 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc"] Sep 10 16:07:48.235561 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:48.235532 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f4b64a3-888f-42b9-be57-c38f5c500873" containerName="payload-pre-processing" Sep 10 16:07:48.235561 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:48.235548 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f4b64a3-888f-42b9-be57-c38f5c500873" containerName="payload-pre-processing" Sep 10 16:07:48.235655 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:48.235568 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a79ea085-ab37-462b-aa56-81f09c2eb640" containerName="payload-processing" Sep 10 16:07:48.235655 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:48.235575 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a79ea085-ab37-462b-aa56-81f09c2eb640" containerName="payload-processing" Sep 10 16:07:48.235655 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:48.235596 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="025fd18f-5774-45e3-b641-db5394c62e12" containerName="payload-pre-processing" Sep 10 16:07:48.235655 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:48.235602 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="025fd18f-5774-45e3-b641-db5394c62e12" containerName="payload-pre-processing" Sep 10 16:07:48.235655 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:48.235609 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c60d8e05-efb7-4e99-8b44-90aac55d3504" containerName="payload-processing" Sep 10 16:07:48.235655 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:48.235614 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c60d8e05-efb7-4e99-8b44-90aac55d3504" containerName="payload-processing" Sep 10 16:07:48.235910 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:48.235677 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a79ea085-ab37-462b-aa56-81f09c2eb640" containerName="payload-processing" Sep 10 16:07:48.235910 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:48.235685 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f4b64a3-888f-42b9-be57-c38f5c500873" containerName="payload-pre-processing" Sep 10 16:07:48.235910 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:48.235693 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="025fd18f-5774-45e3-b641-db5394c62e12" containerName="payload-pre-processing" Sep 10 16:07:48.235910 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:48.235699 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c60d8e05-efb7-4e99-8b44-90aac55d3504" containerName="payload-processing" Sep 10 16:07:48.244276 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:48.244238 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc" Sep 10 16:07:48.247314 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:48.247287 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-e9f330-openshift-default-dockercfg-679wr\"" Sep 10 16:07:48.252488 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:48.252459 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc"] Sep 10 16:07:48.285024 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:48.284975 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:48.386029 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:48.385987 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:48.420239 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:48.420198 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19052e67-3c56-4052-b5be-05e93c3cdd69-credential-socket\") pod \"e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc\" (UID: \"19052e67-3c56-4052-b5be-05e93c3cdd69\") " pod="openshift-ingress/e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc" Sep 10 16:07:48.420445 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:48.420258 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpf4x\" (UniqueName: \"kubernetes.io/projected/19052e67-3c56-4052-b5be-05e93c3cdd69-kube-api-access-tpf4x\") pod \"e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc\" (UID: \"19052e67-3c56-4052-b5be-05e93c3cdd69\") " pod="openshift-ingress/e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc" Sep 10 16:07:48.420445 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:48.420363 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19052e67-3c56-4052-b5be-05e93c3cdd69-istio-data\") pod \"e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc\" (UID: \"19052e67-3c56-4052-b5be-05e93c3cdd69\") " pod="openshift-ingress/e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc" Sep 10 16:07:48.420445 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:48.420393 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19052e67-3c56-4052-b5be-05e93c3cdd69-istio-podinfo\") pod \"e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc\" (UID: \"19052e67-3c56-4052-b5be-05e93c3cdd69\") " pod="openshift-ingress/e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc" Sep 10 16:07:48.420581 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:48.420453 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19052e67-3c56-4052-b5be-05e93c3cdd69-istiod-ca-cert\") pod \"e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc\" (UID: \"19052e67-3c56-4052-b5be-05e93c3cdd69\") " pod="openshift-ingress/e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc" Sep 10 16:07:48.420581 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:48.420473 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19052e67-3c56-4052-b5be-05e93c3cdd69-workload-certs\") pod \"e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc\" (UID: \"19052e67-3c56-4052-b5be-05e93c3cdd69\") " pod="openshift-ingress/e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc" Sep 10 16:07:48.420581 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:48.420520 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19052e67-3c56-4052-b5be-05e93c3cdd69-istio-envoy\") pod \"e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc\" (UID: \"19052e67-3c56-4052-b5be-05e93c3cdd69\") " pod="openshift-ingress/e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc" Sep 10 16:07:48.420581 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:48.420565 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19052e67-3c56-4052-b5be-05e93c3cdd69-istio-token\") pod \"e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc\" (UID: \"19052e67-3c56-4052-b5be-05e93c3cdd69\") " pod="openshift-ingress/e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc" Sep 10 16:07:48.420721 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:48.420602 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19052e67-3c56-4052-b5be-05e93c3cdd69-workload-socket\") pod \"e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc\" (UID: \"19052e67-3c56-4052-b5be-05e93c3cdd69\") " pod="openshift-ingress/e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc" Sep 10 16:07:48.521597 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:48.521487 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19052e67-3c56-4052-b5be-05e93c3cdd69-istiod-ca-cert\") pod \"e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc\" (UID: \"19052e67-3c56-4052-b5be-05e93c3cdd69\") " pod="openshift-ingress/e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc" Sep 10 16:07:48.521597 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:48.521542 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19052e67-3c56-4052-b5be-05e93c3cdd69-workload-certs\") pod \"e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc\" (UID: \"19052e67-3c56-4052-b5be-05e93c3cdd69\") " pod="openshift-ingress/e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc" Sep 10 16:07:48.521597 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:48.521582 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19052e67-3c56-4052-b5be-05e93c3cdd69-istio-envoy\") pod \"e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc\" (UID: \"19052e67-3c56-4052-b5be-05e93c3cdd69\") " pod="openshift-ingress/e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc" Sep 10 16:07:48.521941 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:48.521624 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19052e67-3c56-4052-b5be-05e93c3cdd69-istio-token\") pod \"e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc\" (UID: \"19052e67-3c56-4052-b5be-05e93c3cdd69\") " pod="openshift-ingress/e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc" Sep 10 16:07:48.521941 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:48.521675 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19052e67-3c56-4052-b5be-05e93c3cdd69-workload-socket\") pod \"e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc\" (UID: \"19052e67-3c56-4052-b5be-05e93c3cdd69\") " pod="openshift-ingress/e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc" Sep 10 16:07:48.521941 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:48.521765 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19052e67-3c56-4052-b5be-05e93c3cdd69-credential-socket\") pod \"e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc\" (UID: \"19052e67-3c56-4052-b5be-05e93c3cdd69\") " pod="openshift-ingress/e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc" Sep 10 16:07:48.521941 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:48.521832 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpf4x\" (UniqueName: \"kubernetes.io/projected/19052e67-3c56-4052-b5be-05e93c3cdd69-kube-api-access-tpf4x\") pod \"e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc\" (UID: \"19052e67-3c56-4052-b5be-05e93c3cdd69\") " pod="openshift-ingress/e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc" Sep 10 16:07:48.521941 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:48.521886 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19052e67-3c56-4052-b5be-05e93c3cdd69-istio-data\") pod \"e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc\" (UID: \"19052e67-3c56-4052-b5be-05e93c3cdd69\") " pod="openshift-ingress/e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc" Sep 10 16:07:48.521941 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:48.521918 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19052e67-3c56-4052-b5be-05e93c3cdd69-istio-podinfo\") pod \"e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc\" (UID: \"19052e67-3c56-4052-b5be-05e93c3cdd69\") " pod="openshift-ingress/e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc" Sep 10 16:07:48.522302 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:48.522073 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19052e67-3c56-4052-b5be-05e93c3cdd69-workload-certs\") pod \"e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc\" (UID: \"19052e67-3c56-4052-b5be-05e93c3cdd69\") " pod="openshift-ingress/e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc" Sep 10 16:07:48.522367 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:48.522341 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19052e67-3c56-4052-b5be-05e93c3cdd69-credential-socket\") pod \"e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc\" (UID: \"19052e67-3c56-4052-b5be-05e93c3cdd69\") " pod="openshift-ingress/e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc" Sep 10 16:07:48.522429 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:48.522368 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19052e67-3c56-4052-b5be-05e93c3cdd69-istio-data\") pod \"e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc\" (UID: \"19052e67-3c56-4052-b5be-05e93c3cdd69\") " pod="openshift-ingress/e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc" Sep 10 16:07:48.522429 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:48.522380 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19052e67-3c56-4052-b5be-05e93c3cdd69-istiod-ca-cert\") pod \"e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc\" (UID: \"19052e67-3c56-4052-b5be-05e93c3cdd69\") " pod="openshift-ingress/e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc" Sep 10 16:07:48.522532 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:48.522474 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19052e67-3c56-4052-b5be-05e93c3cdd69-workload-socket\") pod \"e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc\" (UID: \"19052e67-3c56-4052-b5be-05e93c3cdd69\") " pod="openshift-ingress/e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc" Sep 10 16:07:48.524224 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:48.524202 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19052e67-3c56-4052-b5be-05e93c3cdd69-istio-envoy\") pod \"e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc\" (UID: \"19052e67-3c56-4052-b5be-05e93c3cdd69\") " pod="openshift-ingress/e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc" Sep 10 16:07:48.524560 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:48.524539 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19052e67-3c56-4052-b5be-05e93c3cdd69-istio-podinfo\") pod \"e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc\" (UID: \"19052e67-3c56-4052-b5be-05e93c3cdd69\") " pod="openshift-ingress/e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc" Sep 10 16:07:48.530862 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:48.530835 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19052e67-3c56-4052-b5be-05e93c3cdd69-istio-token\") pod \"e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc\" (UID: \"19052e67-3c56-4052-b5be-05e93c3cdd69\") " pod="openshift-ingress/e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc" Sep 10 16:07:48.530982 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:48.530966 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpf4x\" (UniqueName: \"kubernetes.io/projected/19052e67-3c56-4052-b5be-05e93c3cdd69-kube-api-access-tpf4x\") pod \"e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc\" (UID: \"19052e67-3c56-4052-b5be-05e93c3cdd69\") " pod="openshift-ingress/e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc" Sep 10 16:07:48.559307 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:48.559275 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc" Sep 10 16:07:48.837612 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:48.837583 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc"] Sep 10 16:07:48.839953 ip-10-0-139-236 kubenswrapper[2576]: W0910 16:07:48.839924 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19052e67_3c56_4052_b5be_05e93c3cdd69.slice/crio-c46ce460eeebb4b5606901301ce2157ad8ddb483fffea18c5b5cdecb843177e5 WatchSource:0}: Error finding container c46ce460eeebb4b5606901301ce2157ad8ddb483fffea18c5b5cdecb843177e5: Status 404 returned error can't find the container with id c46ce460eeebb4b5606901301ce2157ad8ddb483fffea18c5b5cdecb843177e5 Sep 10 16:07:48.842361 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:48.842325 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:07:48.842446 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:48.842396 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:07:48.842446 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:48.842427 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:07:49.174014 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:49.173913 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f4b64a3-888f-42b9-be57-c38f5c500873" path="/var/lib/kubelet/pods/0f4b64a3-888f-42b9-be57-c38f5c500873/volumes" Sep 10 16:07:49.174427 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:49.174405 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a79ea085-ab37-462b-aa56-81f09c2eb640" path="/var/lib/kubelet/pods/a79ea085-ab37-462b-aa56-81f09c2eb640/volumes" Sep 10 16:07:49.256306 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:49.256268 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc" event={"ID":"19052e67-3c56-4052-b5be-05e93c3cdd69","Type":"ContainerStarted","Data":"61edc4c4d96f3d0ffea786997c6a8857b67bd72d0347664d18adc659d663a0cd"} Sep 10 16:07:49.256767 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:49.256325 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc" event={"ID":"19052e67-3c56-4052-b5be-05e93c3cdd69","Type":"ContainerStarted","Data":"c46ce460eeebb4b5606901301ce2157ad8ddb483fffea18c5b5cdecb843177e5"} Sep 10 16:07:49.281094 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:49.281023 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc" podStartSLOduration=1.281002688 podStartE2EDuration="1.281002688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:07:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:07:49.279658858 +0000 UTC m=+1546.633178053" watchObservedRunningTime="2026-09-10 16:07:49.281002688 +0000 UTC m=+1546.634521928" Sep 10 16:07:49.559627 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:49.559587 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc" Sep 10 16:07:49.565851 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:49.565825 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc" Sep 10 16:07:50.261918 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:50.261886 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc" Sep 10 16:07:50.263059 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:50.263036 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc" Sep 10 16:07:51.285620 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:51.285586 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:51.381825 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:51.381793 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:52.068322 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:52.068287 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:52.196418 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:52.196376 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:52.487040 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:52.486944 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:52.607264 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:52.607223 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:52.901876 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:52.901838 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:53.092881 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:53.092849 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:53.185341 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:53.185228 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:53.288664 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:53.288631 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e9f330-5d469f898b-tz9jl"] Sep 10 16:07:53.293509 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:53.293493 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e9f330-5d469f898b-tz9jl" Sep 10 16:07:53.297038 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:53.297011 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-e9f330\"" Sep 10 16:07:53.297038 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:53.297024 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-e9f330-dockercfg-96n5d\"" Sep 10 16:07:53.318502 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:53.318457 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e9f330-5d469f898b-tz9jl"] Sep 10 16:07:53.474835 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:53.474720 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d80d4de-29ec-4b05-899d-7f9c9e6778be-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e9f330-5d469f898b-tz9jl\" (UID: \"0d80d4de-29ec-4b05-899d-7f9c9e6778be\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e9f330-5d469f898b-tz9jl" Sep 10 16:07:53.474835 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:53.474809 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkpbb\" (UniqueName: \"kubernetes.io/projected/0d80d4de-29ec-4b05-899d-7f9c9e6778be-kube-api-access-jkpbb\") pod \"payload-pre-processing-e2e-ipp-b-e9f330-5d469f898b-tz9jl\" (UID: \"0d80d4de-29ec-4b05-899d-7f9c9e6778be\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e9f330-5d469f898b-tz9jl" Sep 10 16:07:53.575700 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:53.575658 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d80d4de-29ec-4b05-899d-7f9c9e6778be-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e9f330-5d469f898b-tz9jl\" (UID: \"0d80d4de-29ec-4b05-899d-7f9c9e6778be\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e9f330-5d469f898b-tz9jl" Sep 10 16:07:53.576231 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:53.575727 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkpbb\" (UniqueName: \"kubernetes.io/projected/0d80d4de-29ec-4b05-899d-7f9c9e6778be-kube-api-access-jkpbb\") pod \"payload-pre-processing-e2e-ipp-b-e9f330-5d469f898b-tz9jl\" (UID: \"0d80d4de-29ec-4b05-899d-7f9c9e6778be\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e9f330-5d469f898b-tz9jl" Sep 10 16:07:53.576426 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:53.576405 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d80d4de-29ec-4b05-899d-7f9c9e6778be-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e9f330-5d469f898b-tz9jl\" (UID: \"0d80d4de-29ec-4b05-899d-7f9c9e6778be\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e9f330-5d469f898b-tz9jl" Sep 10 16:07:53.585009 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:53.584985 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkpbb\" (UniqueName: \"kubernetes.io/projected/0d80d4de-29ec-4b05-899d-7f9c9e6778be-kube-api-access-jkpbb\") pod \"payload-pre-processing-e2e-ipp-b-e9f330-5d469f898b-tz9jl\" (UID: \"0d80d4de-29ec-4b05-899d-7f9c9e6778be\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e9f330-5d469f898b-tz9jl" Sep 10 16:07:53.604953 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:53.604919 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e9f330-5d469f898b-tz9jl" Sep 10 16:07:53.879713 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:53.879686 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e9f330-5d469f898b-tz9jl"] Sep 10 16:07:53.881489 ip-10-0-139-236 kubenswrapper[2576]: W0910 16:07:53.881445 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d80d4de_29ec_4b05_899d_7f9c9e6778be.slice/crio-44a16486fbfc2c8dcb6c89bee843137389cfed32ff8f19920aa61e4c22998bd7 WatchSource:0}: Error finding container 44a16486fbfc2c8dcb6c89bee843137389cfed32ff8f19920aa61e4c22998bd7: Status 404 returned error can't find the container with id 44a16486fbfc2c8dcb6c89bee843137389cfed32ff8f19920aa61e4c22998bd7 Sep 10 16:07:54.284381 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:54.284281 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e9f330-5d469f898b-tz9jl" event={"ID":"0d80d4de-29ec-4b05-899d-7f9c9e6778be","Type":"ContainerStarted","Data":"0d9c717825a20d3adbb84467c2483b5aac050f7db6199524c44dd90e9f04b23e"} Sep 10 16:07:54.284381 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:54.284342 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e9f330-5d469f898b-tz9jl" event={"ID":"0d80d4de-29ec-4b05-899d-7f9c9e6778be","Type":"ContainerStarted","Data":"44a16486fbfc2c8dcb6c89bee843137389cfed32ff8f19920aa61e4c22998bd7"} Sep 10 16:07:54.284610 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:54.284499 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e9f330-5d469f898b-tz9jl" Sep 10 16:07:54.308383 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:54.308326 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e9f330-5d469f898b-tz9jl" podStartSLOduration=1.308308892 podStartE2EDuration="1.308308892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:07:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:07:54.306414012 +0000 UTC m=+1551.659933197" watchObservedRunningTime="2026-09-10 16:07:54.308308892 +0000 UTC m=+1551.661828113" Sep 10 16:07:54.759800 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:54.759761 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:54.860178 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:54.860146 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:55.012031 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:55.011936 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4"] Sep 10 16:07:55.012212 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:55.012186 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4" podUID="490fdd5a-363b-4314-92a1-23bc1f896c53" containerName="istio-proxy" containerID="cri-o://717f79f978dde50e9f15190a11eee80dfea663990c8b532df0c3ffd74eb434ef" gracePeriod=30 Sep 10 16:07:55.028216 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:55.028176 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:55.476575 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:55.476541 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:56.304568 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:56.304526 2576 patch_prober.go:28] interesting pod/e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 10 16:07:56.305006 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:07:56.304585 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4" podUID="490fdd5a-363b-4314-92a1-23bc1f896c53" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 16:08:00.278869 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:00.278797 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4" Sep 10 16:08:00.321387 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:00.321356 2576 generic.go:356] "Generic (PLEG): container finished" podID="490fdd5a-363b-4314-92a1-23bc1f896c53" containerID="717f79f978dde50e9f15190a11eee80dfea663990c8b532df0c3ffd74eb434ef" exitCode=0 Sep 10 16:08:00.321573 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:00.321420 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4" Sep 10 16:08:00.321573 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:00.321441 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4" event={"ID":"490fdd5a-363b-4314-92a1-23bc1f896c53","Type":"ContainerDied","Data":"717f79f978dde50e9f15190a11eee80dfea663990c8b532df0c3ffd74eb434ef"} Sep 10 16:08:00.321573 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:00.321517 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4" event={"ID":"490fdd5a-363b-4314-92a1-23bc1f896c53","Type":"ContainerDied","Data":"96e29b00d436cb688a063e4f7c2e046ad007399e9b980146999e1c3132c09bef"} Sep 10 16:08:00.321781 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:00.321562 2576 scope.go:117] "RemoveContainer" containerID="717f79f978dde50e9f15190a11eee80dfea663990c8b532df0c3ffd74eb434ef" Sep 10 16:08:00.332175 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:00.332150 2576 scope.go:117] "RemoveContainer" containerID="cf55a4ae2cf420e2ce6cddc67073593fbf01228c61e9bd954ee0b0f82586f2ce" Sep 10 16:08:00.333611 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:00.333469 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/490fdd5a-363b-4314-92a1-23bc1f896c53-istio-data\") pod \"490fdd5a-363b-4314-92a1-23bc1f896c53\" (UID: \"490fdd5a-363b-4314-92a1-23bc1f896c53\") " Sep 10 16:08:00.333611 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:00.333515 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2rgkw\" (UniqueName: \"kubernetes.io/projected/490fdd5a-363b-4314-92a1-23bc1f896c53-kube-api-access-2rgkw\") pod \"490fdd5a-363b-4314-92a1-23bc1f896c53\" (UID: \"490fdd5a-363b-4314-92a1-23bc1f896c53\") " Sep 10 16:08:00.333611 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:00.333565 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/490fdd5a-363b-4314-92a1-23bc1f896c53-workload-socket\") pod \"490fdd5a-363b-4314-92a1-23bc1f896c53\" (UID: \"490fdd5a-363b-4314-92a1-23bc1f896c53\") " Sep 10 16:08:00.333611 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:00.333606 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/490fdd5a-363b-4314-92a1-23bc1f896c53-istiod-ca-cert\") pod \"490fdd5a-363b-4314-92a1-23bc1f896c53\" (UID: \"490fdd5a-363b-4314-92a1-23bc1f896c53\") " Sep 10 16:08:00.333917 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:00.333649 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/490fdd5a-363b-4314-92a1-23bc1f896c53-workload-certs\") pod \"490fdd5a-363b-4314-92a1-23bc1f896c53\" (UID: \"490fdd5a-363b-4314-92a1-23bc1f896c53\") " Sep 10 16:08:00.333917 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:00.333687 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/490fdd5a-363b-4314-92a1-23bc1f896c53-istio-token\") pod \"490fdd5a-363b-4314-92a1-23bc1f896c53\" (UID: \"490fdd5a-363b-4314-92a1-23bc1f896c53\") " Sep 10 16:08:00.333917 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:00.333828 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/490fdd5a-363b-4314-92a1-23bc1f896c53-istio-data" (OuterVolumeSpecName: "istio-data") pod "490fdd5a-363b-4314-92a1-23bc1f896c53" (UID: "490fdd5a-363b-4314-92a1-23bc1f896c53"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:08:00.333917 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:00.333847 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/490fdd5a-363b-4314-92a1-23bc1f896c53-istio-podinfo\") pod \"490fdd5a-363b-4314-92a1-23bc1f896c53\" (UID: \"490fdd5a-363b-4314-92a1-23bc1f896c53\") " Sep 10 16:08:00.334088 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:00.333919 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/490fdd5a-363b-4314-92a1-23bc1f896c53-credential-socket\") pod \"490fdd5a-363b-4314-92a1-23bc1f896c53\" (UID: \"490fdd5a-363b-4314-92a1-23bc1f896c53\") " Sep 10 16:08:00.334088 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:00.333945 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/490fdd5a-363b-4314-92a1-23bc1f896c53-istio-envoy\") pod \"490fdd5a-363b-4314-92a1-23bc1f896c53\" (UID: \"490fdd5a-363b-4314-92a1-23bc1f896c53\") " Sep 10 16:08:00.334202 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:00.334179 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/490fdd5a-363b-4314-92a1-23bc1f896c53-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "490fdd5a-363b-4314-92a1-23bc1f896c53" (UID: "490fdd5a-363b-4314-92a1-23bc1f896c53"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:08:00.334600 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:00.334450 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/490fdd5a-363b-4314-92a1-23bc1f896c53-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "490fdd5a-363b-4314-92a1-23bc1f896c53" (UID: "490fdd5a-363b-4314-92a1-23bc1f896c53"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:08:00.334892 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:00.334855 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/490fdd5a-363b-4314-92a1-23bc1f896c53-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "490fdd5a-363b-4314-92a1-23bc1f896c53" (UID: "490fdd5a-363b-4314-92a1-23bc1f896c53"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:08:00.335157 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:00.335093 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/490fdd5a-363b-4314-92a1-23bc1f896c53-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "490fdd5a-363b-4314-92a1-23bc1f896c53" (UID: "490fdd5a-363b-4314-92a1-23bc1f896c53"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:08:00.335389 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:00.335353 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/490fdd5a-363b-4314-92a1-23bc1f896c53-istiod-ca-cert\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:08:00.335548 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:00.335535 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/490fdd5a-363b-4314-92a1-23bc1f896c53-workload-certs\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:08:00.335656 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:00.335645 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/490fdd5a-363b-4314-92a1-23bc1f896c53-istio-data\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:08:00.336978 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:00.336956 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/490fdd5a-363b-4314-92a1-23bc1f896c53-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "490fdd5a-363b-4314-92a1-23bc1f896c53" (UID: "490fdd5a-363b-4314-92a1-23bc1f896c53"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:08:00.337254 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:00.337232 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/490fdd5a-363b-4314-92a1-23bc1f896c53-kube-api-access-2rgkw" (OuterVolumeSpecName: "kube-api-access-2rgkw") pod "490fdd5a-363b-4314-92a1-23bc1f896c53" (UID: "490fdd5a-363b-4314-92a1-23bc1f896c53"). InnerVolumeSpecName "kube-api-access-2rgkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:08:00.337368 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:00.337254 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/490fdd5a-363b-4314-92a1-23bc1f896c53-istio-token" (OuterVolumeSpecName: "istio-token") pod "490fdd5a-363b-4314-92a1-23bc1f896c53" (UID: "490fdd5a-363b-4314-92a1-23bc1f896c53"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:08:00.337708 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:00.337683 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/490fdd5a-363b-4314-92a1-23bc1f896c53-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "490fdd5a-363b-4314-92a1-23bc1f896c53" (UID: "490fdd5a-363b-4314-92a1-23bc1f896c53"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 16:08:00.357281 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:00.357258 2576 scope.go:117] "RemoveContainer" containerID="717f79f978dde50e9f15190a11eee80dfea663990c8b532df0c3ffd74eb434ef" Sep 10 16:08:00.357687 ip-10-0-139-236 kubenswrapper[2576]: E0910 16:08:00.357662 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"717f79f978dde50e9f15190a11eee80dfea663990c8b532df0c3ffd74eb434ef\": container with ID starting with 717f79f978dde50e9f15190a11eee80dfea663990c8b532df0c3ffd74eb434ef not found: ID does not exist" containerID="717f79f978dde50e9f15190a11eee80dfea663990c8b532df0c3ffd74eb434ef" Sep 10 16:08:00.357800 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:00.357698 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"717f79f978dde50e9f15190a11eee80dfea663990c8b532df0c3ffd74eb434ef"} err="failed to get container status \"717f79f978dde50e9f15190a11eee80dfea663990c8b532df0c3ffd74eb434ef\": rpc error: code = NotFound desc = could not find container \"717f79f978dde50e9f15190a11eee80dfea663990c8b532df0c3ffd74eb434ef\": container with ID starting with 717f79f978dde50e9f15190a11eee80dfea663990c8b532df0c3ffd74eb434ef not found: ID does not exist" Sep 10 16:08:00.357800 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:00.357719 2576 scope.go:117] "RemoveContainer" containerID="cf55a4ae2cf420e2ce6cddc67073593fbf01228c61e9bd954ee0b0f82586f2ce" Sep 10 16:08:00.358201 ip-10-0-139-236 kubenswrapper[2576]: E0910 16:08:00.358180 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf55a4ae2cf420e2ce6cddc67073593fbf01228c61e9bd954ee0b0f82586f2ce\": container with ID starting with cf55a4ae2cf420e2ce6cddc67073593fbf01228c61e9bd954ee0b0f82586f2ce not found: ID does not exist" containerID="cf55a4ae2cf420e2ce6cddc67073593fbf01228c61e9bd954ee0b0f82586f2ce" Sep 10 16:08:00.358300 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:00.358206 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf55a4ae2cf420e2ce6cddc67073593fbf01228c61e9bd954ee0b0f82586f2ce"} err="failed to get container status \"cf55a4ae2cf420e2ce6cddc67073593fbf01228c61e9bd954ee0b0f82586f2ce\": rpc error: code = NotFound desc = could not find container \"cf55a4ae2cf420e2ce6cddc67073593fbf01228c61e9bd954ee0b0f82586f2ce\": container with ID starting with cf55a4ae2cf420e2ce6cddc67073593fbf01228c61e9bd954ee0b0f82586f2ce not found: ID does not exist" Sep 10 16:08:00.436524 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:00.436479 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/490fdd5a-363b-4314-92a1-23bc1f896c53-credential-socket\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:08:00.436524 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:00.436516 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/490fdd5a-363b-4314-92a1-23bc1f896c53-istio-envoy\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:08:00.436524 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:00.436526 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rgkw\" (UniqueName: \"kubernetes.io/projected/490fdd5a-363b-4314-92a1-23bc1f896c53-kube-api-access-2rgkw\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:08:00.436524 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:00.436535 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/490fdd5a-363b-4314-92a1-23bc1f896c53-workload-socket\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:08:00.436888 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:00.436544 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/490fdd5a-363b-4314-92a1-23bc1f896c53-istio-token\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:08:00.436888 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:00.436553 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/490fdd5a-363b-4314-92a1-23bc1f896c53-istio-podinfo\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:08:00.649801 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:00.649764 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4"] Sep 10 16:08:00.653945 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:00.653908 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a361a9-openshift-default-6679498d4-fghw4"] Sep 10 16:08:01.092518 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:01.092478 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 16:08:01.171285 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:01.171248 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="490fdd5a-363b-4314-92a1-23bc1f896c53" path="/var/lib/kubelet/pods/490fdd5a-363b-4314-92a1-23bc1f896c53/volumes" Sep 10 16:08:05.292373 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:05.292338 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e9f330-5d469f898b-tz9jl" Sep 10 16:08:05.383522 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:05.383485 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:08:05.986163 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:05.986123 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:08:06.027575 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:06.027534 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-33451c-857779b665-hdg6c"] Sep 10 16:08:06.027866 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:06.027806 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-33451c-857779b665-hdg6c" podUID="a1b7117e-21cf-47b0-a845-0f7b7b78ecca" containerName="payload-processing" containerID="cri-o://8d8e4d1cab32d192a54e0a81096eb9601e91e314a4dea3f3bd989df6960059d5" gracePeriod=30 Sep 10 16:08:06.040290 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:06.040257 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-33451c-76474cff66-qwn9d"] Sep 10 16:08:06.040590 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:06.040509 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-33451c-76474cff66-qwn9d" podUID="93edc94e-43ca-4cf9-9716-b2478310d631" containerName="payload-pre-processing" containerID="cri-o://6851b686d21b22949d9ba682020ea07b976d2d82efda071f961d740408564e20" gracePeriod=30 Sep 10 16:08:06.081349 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:06.081306 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:08:06.157065 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:06.157031 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:08:06.240891 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:06.240550 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:08:06.355553 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:06.355526 2576 generic.go:356] "Generic (PLEG): container finished" podID="93edc94e-43ca-4cf9-9716-b2478310d631" containerID="6851b686d21b22949d9ba682020ea07b976d2d82efda071f961d740408564e20" exitCode=0 Sep 10 16:08:06.355940 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:06.355608 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-33451c-76474cff66-qwn9d" event={"ID":"93edc94e-43ca-4cf9-9716-b2478310d631","Type":"ContainerDied","Data":"6851b686d21b22949d9ba682020ea07b976d2d82efda071f961d740408564e20"} Sep 10 16:08:06.357274 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:06.357250 2576 generic.go:356] "Generic (PLEG): container finished" podID="a1b7117e-21cf-47b0-a845-0f7b7b78ecca" containerID="8d8e4d1cab32d192a54e0a81096eb9601e91e314a4dea3f3bd989df6960059d5" exitCode=0 Sep 10 16:08:06.357384 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:06.357317 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-33451c-857779b665-hdg6c" event={"ID":"a1b7117e-21cf-47b0-a845-0f7b7b78ecca","Type":"ContainerDied","Data":"8d8e4d1cab32d192a54e0a81096eb9601e91e314a4dea3f3bd989df6960059d5"} Sep 10 16:08:06.372227 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:06.372193 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-33451c-857779b665-hdg6c" Sep 10 16:08:06.396058 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:06.396013 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cccm6\" (UniqueName: \"kubernetes.io/projected/a1b7117e-21cf-47b0-a845-0f7b7b78ecca-kube-api-access-cccm6\") pod \"a1b7117e-21cf-47b0-a845-0f7b7b78ecca\" (UID: \"a1b7117e-21cf-47b0-a845-0f7b7b78ecca\") " Sep 10 16:08:06.396235 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:06.396081 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1b7117e-21cf-47b0-a845-0f7b7b78ecca-plugins-config-volume\") pod \"a1b7117e-21cf-47b0-a845-0f7b7b78ecca\" (UID: \"a1b7117e-21cf-47b0-a845-0f7b7b78ecca\") " Sep 10 16:08:06.422767 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:06.405007 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1b7117e-21cf-47b0-a845-0f7b7b78ecca-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a1b7117e-21cf-47b0-a845-0f7b7b78ecca" (UID: "a1b7117e-21cf-47b0-a845-0f7b7b78ecca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:08:06.422767 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:06.408879 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1b7117e-21cf-47b0-a845-0f7b7b78ecca-kube-api-access-cccm6" (OuterVolumeSpecName: "kube-api-access-cccm6") pod "a1b7117e-21cf-47b0-a845-0f7b7b78ecca" (UID: "a1b7117e-21cf-47b0-a845-0f7b7b78ecca"). InnerVolumeSpecName "kube-api-access-cccm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:08:06.422767 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:06.410919 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-33451c-76474cff66-qwn9d" Sep 10 16:08:06.511815 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:06.504058 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lz7cz\" (UniqueName: \"kubernetes.io/projected/93edc94e-43ca-4cf9-9716-b2478310d631-kube-api-access-lz7cz\") pod \"93edc94e-43ca-4cf9-9716-b2478310d631\" (UID: \"93edc94e-43ca-4cf9-9716-b2478310d631\") " Sep 10 16:08:06.511815 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:06.504128 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93edc94e-43ca-4cf9-9716-b2478310d631-plugins-config-volume\") pod \"93edc94e-43ca-4cf9-9716-b2478310d631\" (UID: \"93edc94e-43ca-4cf9-9716-b2478310d631\") " Sep 10 16:08:06.511815 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:06.504575 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cccm6\" (UniqueName: \"kubernetes.io/projected/a1b7117e-21cf-47b0-a845-0f7b7b78ecca-kube-api-access-cccm6\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:08:06.511815 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:06.504599 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1b7117e-21cf-47b0-a845-0f7b7b78ecca-plugins-config-volume\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:08:06.511815 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:06.509701 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93edc94e-43ca-4cf9-9716-b2478310d631-kube-api-access-lz7cz" (OuterVolumeSpecName: "kube-api-access-lz7cz") pod "93edc94e-43ca-4cf9-9716-b2478310d631" (UID: "93edc94e-43ca-4cf9-9716-b2478310d631"). InnerVolumeSpecName "kube-api-access-lz7cz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:08:06.511815 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:06.509948 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93edc94e-43ca-4cf9-9716-b2478310d631-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "93edc94e-43ca-4cf9-9716-b2478310d631" (UID: "93edc94e-43ca-4cf9-9716-b2478310d631"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:08:06.594606 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:06.594573 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:08:06.606081 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:06.606035 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lz7cz\" (UniqueName: \"kubernetes.io/projected/93edc94e-43ca-4cf9-9716-b2478310d631-kube-api-access-lz7cz\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:08:06.606081 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:06.606079 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93edc94e-43ca-4cf9-9716-b2478310d631-plugins-config-volume\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:08:07.364642 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:07.364550 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-33451c-857779b665-hdg6c" Sep 10 16:08:07.364642 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:07.364583 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-33451c-857779b665-hdg6c" event={"ID":"a1b7117e-21cf-47b0-a845-0f7b7b78ecca","Type":"ContainerDied","Data":"787f05e67faa6ac092fd04e7585eb1892df568c73a183c5bfe721630fbffb27a"} Sep 10 16:08:07.365193 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:07.364651 2576 scope.go:117] "RemoveContainer" containerID="8d8e4d1cab32d192a54e0a81096eb9601e91e314a4dea3f3bd989df6960059d5" Sep 10 16:08:07.366065 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:07.366019 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-33451c-76474cff66-qwn9d" event={"ID":"93edc94e-43ca-4cf9-9716-b2478310d631","Type":"ContainerDied","Data":"b05c05bda589b5b998b544cb24e0af3c8b2f84f921f67319a0af116c20bf594c"} Sep 10 16:08:07.366065 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:07.366029 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-33451c-76474cff66-qwn9d" Sep 10 16:08:07.373821 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:07.373800 2576 scope.go:117] "RemoveContainer" containerID="6851b686d21b22949d9ba682020ea07b976d2d82efda071f961d740408564e20" Sep 10 16:08:07.386483 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:07.386425 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-33451c-76474cff66-qwn9d"] Sep 10 16:08:07.390745 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:07.390715 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-33451c-76474cff66-qwn9d"] Sep 10 16:08:07.410222 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:07.410181 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-33451c-857779b665-hdg6c"] Sep 10 16:08:07.414800 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:07.414767 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-33451c-857779b665-hdg6c"] Sep 10 16:08:09.170545 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:09.170505 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93edc94e-43ca-4cf9-9716-b2478310d631" path="/var/lib/kubelet/pods/93edc94e-43ca-4cf9-9716-b2478310d631/volumes" Sep 10 16:08:09.170969 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:09.170929 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1b7117e-21cf-47b0-a845-0f7b7b78ecca" path="/var/lib/kubelet/pods/a1b7117e-21cf-47b0-a845-0f7b7b78ecca/volumes" Sep 10 16:08:11.392495 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:11.392465 2576 generic.go:356] "Generic (PLEG): container finished" podID="19052e67-3c56-4052-b5be-05e93c3cdd69" containerID="61edc4c4d96f3d0ffea786997c6a8857b67bd72d0347664d18adc659d663a0cd" exitCode=0 Sep 10 16:08:11.392928 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:11.392521 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc" event={"ID":"19052e67-3c56-4052-b5be-05e93c3cdd69","Type":"ContainerDied","Data":"61edc4c4d96f3d0ffea786997c6a8857b67bd72d0347664d18adc659d663a0cd"} Sep 10 16:08:11.392979 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:11.392963 2576 scope.go:117] "RemoveContainer" containerID="61edc4c4d96f3d0ffea786997c6a8857b67bd72d0347664d18adc659d663a0cd" Sep 10 16:08:11.394165 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:11.394134 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:08:11.394235 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:11.394209 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:08:11.394277 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:11.394242 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:08:11.893327 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:11.893292 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 16:08:12.403462 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:12.403426 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc" event={"ID":"19052e67-3c56-4052-b5be-05e93c3cdd69","Type":"ContainerStarted","Data":"2dc3f7d130f1f360127d4453c7809c0a98c5c5996ea93318c153a3bc3c12515f"} Sep 10 16:08:12.559462 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:12.559428 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc" Sep 10 16:08:12.559687 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:12.559666 2576 patch_prober.go:28] interesting pod/e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Sep 10 16:08:12.559778 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:12.559759 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc" podUID="19052e67-3c56-4052-b5be-05e93c3cdd69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Sep 10 16:08:13.515601 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:13.515562 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:08:13.559668 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:13.559636 2576 patch_prober.go:28] interesting pod/e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Sep 10 16:08:13.559923 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:13.559689 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc" podUID="19052e67-3c56-4052-b5be-05e93c3cdd69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Sep 10 16:08:13.606793 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:13.606696 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:08:13.719192 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:13.719142 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:08:13.809073 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:13.809029 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:08:14.560677 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:14.560638 2576 patch_prober.go:28] interesting pod/e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Sep 10 16:08:14.561123 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:14.560702 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc" podUID="19052e67-3c56-4052-b5be-05e93c3cdd69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Sep 10 16:08:15.560683 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:15.560644 2576 patch_prober.go:28] interesting pod/e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Sep 10 16:08:15.561120 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:15.560717 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc" podUID="19052e67-3c56-4052-b5be-05e93c3cdd69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Sep 10 16:08:16.136206 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:16.136165 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:08:16.560023 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:16.559986 2576 patch_prober.go:28] interesting pod/e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Sep 10 16:08:16.560228 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:16.560056 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc" podUID="19052e67-3c56-4052-b5be-05e93c3cdd69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Sep 10 16:08:17.560370 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:17.560318 2576 patch_prober.go:28] interesting pod/e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Sep 10 16:08:17.560890 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:17.560401 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc" podUID="19052e67-3c56-4052-b5be-05e93c3cdd69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Sep 10 16:08:18.282217 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:18.282179 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:08:18.564338 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:18.564306 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc" Sep 10 16:08:18.564829 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:18.564609 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc" Sep 10 16:08:18.565365 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:18.565348 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc" Sep 10 16:08:30.314379 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:30.314344 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:08:33.193409 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:33.193362 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:08:37.302767 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:37.302693 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 16:08:50.788289 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:50.788246 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 16:08:56.948727 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:56.948646 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-c0ec41-69664b596f-642w8_168d3721-ce5b-42b7-b3c5-44106c77ac3f/payload-processing/0.log" Sep 10 16:08:57.533240 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:57.533194 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:08:57.824406 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:57.824370 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4"] Sep 10 16:08:57.825093 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:57.825074 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="490fdd5a-363b-4314-92a1-23bc1f896c53" containerName="istio-proxy" Sep 10 16:08:57.825149 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:57.825099 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="490fdd5a-363b-4314-92a1-23bc1f896c53" containerName="istio-proxy" Sep 10 16:08:57.825149 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:57.825129 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93edc94e-43ca-4cf9-9716-b2478310d631" containerName="payload-pre-processing" Sep 10 16:08:57.825149 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:57.825138 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="93edc94e-43ca-4cf9-9716-b2478310d631" containerName="payload-pre-processing" Sep 10 16:08:57.825246 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:57.825151 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1b7117e-21cf-47b0-a845-0f7b7b78ecca" containerName="payload-processing" Sep 10 16:08:57.825246 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:57.825161 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1b7117e-21cf-47b0-a845-0f7b7b78ecca" containerName="payload-processing" Sep 10 16:08:57.825246 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:57.825178 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="490fdd5a-363b-4314-92a1-23bc1f896c53" containerName="istio-proxy" Sep 10 16:08:57.825246 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:57.825187 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="490fdd5a-363b-4314-92a1-23bc1f896c53" containerName="istio-proxy" Sep 10 16:08:57.825370 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:57.825280 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="490fdd5a-363b-4314-92a1-23bc1f896c53" containerName="istio-proxy" Sep 10 16:08:57.825370 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:57.825293 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1b7117e-21cf-47b0-a845-0f7b7b78ecca" containerName="payload-processing" Sep 10 16:08:57.825370 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:57.825303 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="93edc94e-43ca-4cf9-9716-b2478310d631" containerName="payload-pre-processing" Sep 10 16:08:57.825370 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:57.825314 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="490fdd5a-363b-4314-92a1-23bc1f896c53" containerName="istio-proxy" Sep 10 16:08:57.829209 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:57.829187 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4" Sep 10 16:08:57.832162 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:57.832135 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-044fdb-openshift-default-dockercfg-nrdwn\"" Sep 10 16:08:57.849525 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:57.849482 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4"] Sep 10 16:08:57.863515 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:57.863481 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:08:57.940876 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:57.940837 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b18da87-bf73-4749-9af5-9c8472c9a962-workload-certs\") pod \"e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4\" (UID: \"0b18da87-bf73-4749-9af5-9c8472c9a962\") " pod="openshift-ingress/e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4" Sep 10 16:08:57.941065 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:57.940909 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b18da87-bf73-4749-9af5-9c8472c9a962-istio-podinfo\") pod \"e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4\" (UID: \"0b18da87-bf73-4749-9af5-9c8472c9a962\") " pod="openshift-ingress/e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4" Sep 10 16:08:57.941065 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:57.940935 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b18da87-bf73-4749-9af5-9c8472c9a962-credential-socket\") pod \"e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4\" (UID: \"0b18da87-bf73-4749-9af5-9c8472c9a962\") " pod="openshift-ingress/e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4" Sep 10 16:08:57.941065 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:57.940964 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b18da87-bf73-4749-9af5-9c8472c9a962-istio-data\") pod \"e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4\" (UID: \"0b18da87-bf73-4749-9af5-9c8472c9a962\") " pod="openshift-ingress/e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4" Sep 10 16:08:57.941173 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:57.941091 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b18da87-bf73-4749-9af5-9c8472c9a962-workload-socket\") pod \"e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4\" (UID: \"0b18da87-bf73-4749-9af5-9c8472c9a962\") " pod="openshift-ingress/e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4" Sep 10 16:08:57.941173 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:57.941123 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b18da87-bf73-4749-9af5-9c8472c9a962-istio-token\") pod \"e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4\" (UID: \"0b18da87-bf73-4749-9af5-9c8472c9a962\") " pod="openshift-ingress/e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4" Sep 10 16:08:57.941243 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:57.941202 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b18da87-bf73-4749-9af5-9c8472c9a962-istio-envoy\") pod \"e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4\" (UID: \"0b18da87-bf73-4749-9af5-9c8472c9a962\") " pod="openshift-ingress/e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4" Sep 10 16:08:57.941243 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:57.941226 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b18da87-bf73-4749-9af5-9c8472c9a962-istiod-ca-cert\") pod \"e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4\" (UID: \"0b18da87-bf73-4749-9af5-9c8472c9a962\") " pod="openshift-ingress/e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4" Sep 10 16:08:57.941312 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:57.941245 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f59p8\" (UniqueName: \"kubernetes.io/projected/0b18da87-bf73-4749-9af5-9c8472c9a962-kube-api-access-f59p8\") pod \"e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4\" (UID: \"0b18da87-bf73-4749-9af5-9c8472c9a962\") " pod="openshift-ingress/e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4" Sep 10 16:08:58.042587 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:58.042546 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b18da87-bf73-4749-9af5-9c8472c9a962-istio-podinfo\") pod \"e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4\" (UID: \"0b18da87-bf73-4749-9af5-9c8472c9a962\") " pod="openshift-ingress/e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4" Sep 10 16:08:58.042587 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:58.042591 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b18da87-bf73-4749-9af5-9c8472c9a962-credential-socket\") pod \"e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4\" (UID: \"0b18da87-bf73-4749-9af5-9c8472c9a962\") " pod="openshift-ingress/e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4" Sep 10 16:08:58.043252 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:58.042610 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b18da87-bf73-4749-9af5-9c8472c9a962-istio-data\") pod \"e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4\" (UID: \"0b18da87-bf73-4749-9af5-9c8472c9a962\") " pod="openshift-ingress/e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4" Sep 10 16:08:58.043252 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:58.042690 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b18da87-bf73-4749-9af5-9c8472c9a962-workload-socket\") pod \"e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4\" (UID: \"0b18da87-bf73-4749-9af5-9c8472c9a962\") " pod="openshift-ingress/e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4" Sep 10 16:08:58.043252 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:58.042709 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b18da87-bf73-4749-9af5-9c8472c9a962-istio-token\") pod \"e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4\" (UID: \"0b18da87-bf73-4749-9af5-9c8472c9a962\") " pod="openshift-ingress/e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4" Sep 10 16:08:58.043252 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:58.042818 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b18da87-bf73-4749-9af5-9c8472c9a962-istio-envoy\") pod \"e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4\" (UID: \"0b18da87-bf73-4749-9af5-9c8472c9a962\") " pod="openshift-ingress/e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4" Sep 10 16:08:58.043252 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:58.042844 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b18da87-bf73-4749-9af5-9c8472c9a962-istiod-ca-cert\") pod \"e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4\" (UID: \"0b18da87-bf73-4749-9af5-9c8472c9a962\") " pod="openshift-ingress/e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4" Sep 10 16:08:58.043252 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:58.042872 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f59p8\" (UniqueName: \"kubernetes.io/projected/0b18da87-bf73-4749-9af5-9c8472c9a962-kube-api-access-f59p8\") pod \"e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4\" (UID: \"0b18da87-bf73-4749-9af5-9c8472c9a962\") " pod="openshift-ingress/e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4" Sep 10 16:08:58.043252 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:58.042890 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b18da87-bf73-4749-9af5-9c8472c9a962-workload-certs\") pod \"e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4\" (UID: \"0b18da87-bf73-4749-9af5-9c8472c9a962\") " pod="openshift-ingress/e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4" Sep 10 16:08:58.043252 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:58.043108 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b18da87-bf73-4749-9af5-9c8472c9a962-istio-data\") pod \"e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4\" (UID: \"0b18da87-bf73-4749-9af5-9c8472c9a962\") " pod="openshift-ingress/e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4" Sep 10 16:08:58.043252 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:58.043238 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b18da87-bf73-4749-9af5-9c8472c9a962-workload-certs\") pod \"e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4\" (UID: \"0b18da87-bf73-4749-9af5-9c8472c9a962\") " pod="openshift-ingress/e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4" Sep 10 16:08:58.043697 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:58.043430 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b18da87-bf73-4749-9af5-9c8472c9a962-workload-socket\") pod \"e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4\" (UID: \"0b18da87-bf73-4749-9af5-9c8472c9a962\") " pod="openshift-ingress/e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4" Sep 10 16:08:58.043697 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:58.043516 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b18da87-bf73-4749-9af5-9c8472c9a962-istiod-ca-cert\") pod \"e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4\" (UID: \"0b18da87-bf73-4749-9af5-9c8472c9a962\") " pod="openshift-ingress/e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4" Sep 10 16:08:58.043697 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:58.043567 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b18da87-bf73-4749-9af5-9c8472c9a962-credential-socket\") pod \"e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4\" (UID: \"0b18da87-bf73-4749-9af5-9c8472c9a962\") " pod="openshift-ingress/e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4" Sep 10 16:08:58.045264 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:58.045233 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b18da87-bf73-4749-9af5-9c8472c9a962-istio-envoy\") pod \"e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4\" (UID: \"0b18da87-bf73-4749-9af5-9c8472c9a962\") " pod="openshift-ingress/e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4" Sep 10 16:08:58.045384 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:58.045339 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b18da87-bf73-4749-9af5-9c8472c9a962-istio-podinfo\") pod \"e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4\" (UID: \"0b18da87-bf73-4749-9af5-9c8472c9a962\") " pod="openshift-ingress/e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4" Sep 10 16:08:58.054267 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:58.054232 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f59p8\" (UniqueName: \"kubernetes.io/projected/0b18da87-bf73-4749-9af5-9c8472c9a962-kube-api-access-f59p8\") pod \"e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4\" (UID: \"0b18da87-bf73-4749-9af5-9c8472c9a962\") " pod="openshift-ingress/e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4" Sep 10 16:08:58.055263 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:58.055247 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b18da87-bf73-4749-9af5-9c8472c9a962-istio-token\") pod \"e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4\" (UID: \"0b18da87-bf73-4749-9af5-9c8472c9a962\") " pod="openshift-ingress/e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4" Sep 10 16:08:58.141831 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:58.141709 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4" Sep 10 16:08:58.177301 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:58.177263 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:08:58.431127 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:58.431096 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4"] Sep 10 16:08:58.434746 ip-10-0-139-236 kubenswrapper[2576]: W0910 16:08:58.434691 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b18da87_bf73_4749_9af5_9c8472c9a962.slice/crio-4f50c225de364bff00ab6bbc693c9eaf03fd4ebf886acc3b316eb6afe3fbdb3d WatchSource:0}: Error finding container 4f50c225de364bff00ab6bbc693c9eaf03fd4ebf886acc3b316eb6afe3fbdb3d: Status 404 returned error can't find the container with id 4f50c225de364bff00ab6bbc693c9eaf03fd4ebf886acc3b316eb6afe3fbdb3d Sep 10 16:08:58.438545 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:58.438506 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:08:58.438658 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:58.438589 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:08:58.438658 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:58.438621 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:08:58.657960 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:58.657921 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4" event={"ID":"0b18da87-bf73-4749-9af5-9c8472c9a962","Type":"ContainerStarted","Data":"5cb5ea6920fbe133a7f74b17a0319ece9ea064edea4b69623b6cfd470ccc9263"} Sep 10 16:08:58.658147 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:58.657984 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4" event={"ID":"0b18da87-bf73-4749-9af5-9c8472c9a962","Type":"ContainerStarted","Data":"4f50c225de364bff00ab6bbc693c9eaf03fd4ebf886acc3b316eb6afe3fbdb3d"} Sep 10 16:08:58.683570 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:58.683438 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4" podStartSLOduration=1.683415001 podStartE2EDuration="1.683415001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:08:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:08:58.68208304 +0000 UTC m=+1616.035602225" watchObservedRunningTime="2026-09-10 16:08:58.683415001 +0000 UTC m=+1616.036934164" Sep 10 16:08:59.141923 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:59.141884 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4" Sep 10 16:08:59.147948 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:59.147918 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4" Sep 10 16:08:59.664542 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:59.664504 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4" Sep 10 16:08:59.665501 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:08:59.665480 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4" Sep 10 16:09:01.615106 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:01.615058 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:09:01.715975 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:01.715941 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:09:01.964754 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:01.964622 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:09:02.067745 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:02.067693 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:09:02.200183 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:02.200139 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:09:02.526583 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:02.526543 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:09:02.626564 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:02.626532 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:09:02.731816 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:02.731776 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-044fdb-76bc9bcd5d-q2jnt"] Sep 10 16:09:02.735477 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:02.735457 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-044fdb-76bc9bcd5d-q2jnt" Sep 10 16:09:02.738588 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:02.738563 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-044fdb-dockercfg-xft4g\"" Sep 10 16:09:02.739004 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:02.738986 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-044fdb\"" Sep 10 16:09:02.762254 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:02.762215 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-044fdb-76bc9bcd5d-q2jnt"] Sep 10 16:09:02.794549 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:02.794504 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b230b042-b3ed-4599-994e-0f698ab9569b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-044fdb-76bc9bcd5d-q2jnt\" (UID: \"b230b042-b3ed-4599-994e-0f698ab9569b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-044fdb-76bc9bcd5d-q2jnt" Sep 10 16:09:02.794808 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:02.794556 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7xtw\" (UniqueName: \"kubernetes.io/projected/b230b042-b3ed-4599-994e-0f698ab9569b-kube-api-access-v7xtw\") pod \"payload-pre-processing-e2e-ipp-cleanup-044fdb-76bc9bcd5d-q2jnt\" (UID: \"b230b042-b3ed-4599-994e-0f698ab9569b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-044fdb-76bc9bcd5d-q2jnt" Sep 10 16:09:02.812215 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:02.812177 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-044fdb-5898c975bc-cb7vn"] Sep 10 16:09:02.816845 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:02.816777 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-044fdb-5898c975bc-cb7vn" Sep 10 16:09:02.835402 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:02.835367 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-044fdb-5898c975bc-cb7vn"] Sep 10 16:09:02.895170 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:02.895135 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b230b042-b3ed-4599-994e-0f698ab9569b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-044fdb-76bc9bcd5d-q2jnt\" (UID: \"b230b042-b3ed-4599-994e-0f698ab9569b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-044fdb-76bc9bcd5d-q2jnt" Sep 10 16:09:02.895170 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:02.895171 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7xtw\" (UniqueName: \"kubernetes.io/projected/b230b042-b3ed-4599-994e-0f698ab9569b-kube-api-access-v7xtw\") pod \"payload-pre-processing-e2e-ipp-cleanup-044fdb-76bc9bcd5d-q2jnt\" (UID: \"b230b042-b3ed-4599-994e-0f698ab9569b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-044fdb-76bc9bcd5d-q2jnt" Sep 10 16:09:02.895455 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:02.895198 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx44v\" (UniqueName: \"kubernetes.io/projected/11736f45-7a39-4b92-b65f-2f7d3a2c5ed9-kube-api-access-gx44v\") pod \"payload-processing-e2e-ipp-cleanup-044fdb-5898c975bc-cb7vn\" (UID: \"11736f45-7a39-4b92-b65f-2f7d3a2c5ed9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-044fdb-5898c975bc-cb7vn" Sep 10 16:09:02.895455 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:02.895252 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11736f45-7a39-4b92-b65f-2f7d3a2c5ed9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-044fdb-5898c975bc-cb7vn\" (UID: \"11736f45-7a39-4b92-b65f-2f7d3a2c5ed9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-044fdb-5898c975bc-cb7vn" Sep 10 16:09:02.896140 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:02.896109 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b230b042-b3ed-4599-994e-0f698ab9569b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-044fdb-76bc9bcd5d-q2jnt\" (UID: \"b230b042-b3ed-4599-994e-0f698ab9569b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-044fdb-76bc9bcd5d-q2jnt" Sep 10 16:09:02.908186 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:02.908157 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7xtw\" (UniqueName: \"kubernetes.io/projected/b230b042-b3ed-4599-994e-0f698ab9569b-kube-api-access-v7xtw\") pod \"payload-pre-processing-e2e-ipp-cleanup-044fdb-76bc9bcd5d-q2jnt\" (UID: \"b230b042-b3ed-4599-994e-0f698ab9569b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-044fdb-76bc9bcd5d-q2jnt" Sep 10 16:09:02.996605 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:02.996542 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11736f45-7a39-4b92-b65f-2f7d3a2c5ed9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-044fdb-5898c975bc-cb7vn\" (UID: \"11736f45-7a39-4b92-b65f-2f7d3a2c5ed9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-044fdb-5898c975bc-cb7vn" Sep 10 16:09:02.996854 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:02.996706 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx44v\" (UniqueName: \"kubernetes.io/projected/11736f45-7a39-4b92-b65f-2f7d3a2c5ed9-kube-api-access-gx44v\") pod \"payload-processing-e2e-ipp-cleanup-044fdb-5898c975bc-cb7vn\" (UID: \"11736f45-7a39-4b92-b65f-2f7d3a2c5ed9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-044fdb-5898c975bc-cb7vn" Sep 10 16:09:02.997400 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:02.997375 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11736f45-7a39-4b92-b65f-2f7d3a2c5ed9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-044fdb-5898c975bc-cb7vn\" (UID: \"11736f45-7a39-4b92-b65f-2f7d3a2c5ed9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-044fdb-5898c975bc-cb7vn" Sep 10 16:09:03.005782 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:03.005723 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx44v\" (UniqueName: \"kubernetes.io/projected/11736f45-7a39-4b92-b65f-2f7d3a2c5ed9-kube-api-access-gx44v\") pod \"payload-processing-e2e-ipp-cleanup-044fdb-5898c975bc-cb7vn\" (UID: \"11736f45-7a39-4b92-b65f-2f7d3a2c5ed9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-044fdb-5898c975bc-cb7vn" Sep 10 16:09:03.046802 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:03.046694 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-044fdb-76bc9bcd5d-q2jnt" Sep 10 16:09:03.132622 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:03.132588 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-044fdb-5898c975bc-cb7vn" Sep 10 16:09:03.325232 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:03.325202 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-044fdb-76bc9bcd5d-q2jnt"] Sep 10 16:09:03.326852 ip-10-0-139-236 kubenswrapper[2576]: W0910 16:09:03.326817 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb230b042_b3ed_4599_994e_0f698ab9569b.slice/crio-ec887868fd0b68849cf0863bb91747ef6709fad124f99df1e11269e9d4625e47 WatchSource:0}: Error finding container ec887868fd0b68849cf0863bb91747ef6709fad124f99df1e11269e9d4625e47: Status 404 returned error can't find the container with id ec887868fd0b68849cf0863bb91747ef6709fad124f99df1e11269e9d4625e47 Sep 10 16:09:03.412666 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:03.412638 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-044fdb-5898c975bc-cb7vn"] Sep 10 16:09:03.414957 ip-10-0-139-236 kubenswrapper[2576]: W0910 16:09:03.414930 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11736f45_7a39_4b92_b65f_2f7d3a2c5ed9.slice/crio-a8772317b74ffa93058d5eb0f9d41a54aa70be7995913e1c90638fe025e67731 WatchSource:0}: Error finding container a8772317b74ffa93058d5eb0f9d41a54aa70be7995913e1c90638fe025e67731: Status 404 returned error can't find the container with id a8772317b74ffa93058d5eb0f9d41a54aa70be7995913e1c90638fe025e67731 Sep 10 16:09:03.696849 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:03.696718 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-044fdb-5898c975bc-cb7vn" event={"ID":"11736f45-7a39-4b92-b65f-2f7d3a2c5ed9","Type":"ContainerStarted","Data":"6b71c0c75f1493b3d0ad7d8e22f9dc8d6081a454bf0667afc31b42755ac459eb"} Sep 10 16:09:03.696849 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:03.696808 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-044fdb-5898c975bc-cb7vn" Sep 10 16:09:03.696849 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:03.696834 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-044fdb-5898c975bc-cb7vn" event={"ID":"11736f45-7a39-4b92-b65f-2f7d3a2c5ed9","Type":"ContainerStarted","Data":"a8772317b74ffa93058d5eb0f9d41a54aa70be7995913e1c90638fe025e67731"} Sep 10 16:09:03.699051 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:03.699023 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-044fdb-76bc9bcd5d-q2jnt" event={"ID":"b230b042-b3ed-4599-994e-0f698ab9569b","Type":"ContainerStarted","Data":"dd52fb15084aa49f7ef4010d0eea8e95adf2a40defaf3cfc2ab84eb6ed75118f"} Sep 10 16:09:03.699221 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:03.699073 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-044fdb-76bc9bcd5d-q2jnt" event={"ID":"b230b042-b3ed-4599-994e-0f698ab9569b","Type":"ContainerStarted","Data":"ec887868fd0b68849cf0863bb91747ef6709fad124f99df1e11269e9d4625e47"} Sep 10 16:09:03.699221 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:03.699180 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-044fdb-76bc9bcd5d-q2jnt" Sep 10 16:09:03.730107 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:03.730047 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-044fdb-5898c975bc-cb7vn" podStartSLOduration=1.730027918 podStartE2EDuration="1.730027918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:09:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:09:03.726047856 +0000 UTC m=+1621.079567045" watchObservedRunningTime="2026-09-10 16:09:03.730027918 +0000 UTC m=+1621.083547084" Sep 10 16:09:03.747437 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:03.747378 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-044fdb-76bc9bcd5d-q2jnt" podStartSLOduration=1.747358677 podStartE2EDuration="1.747358677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:09:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:09:03.744788765 +0000 UTC m=+1621.098307937" watchObservedRunningTime="2026-09-10 16:09:03.747358677 +0000 UTC m=+1621.100877838" Sep 10 16:09:10.491037 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:10.490998 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 16:09:14.708108 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:14.708063 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-044fdb-5898c975bc-cb7vn" Sep 10 16:09:14.710895 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:14.710861 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-044fdb-76bc9bcd5d-q2jnt" Sep 10 16:09:21.800016 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:21.799971 2576 generic.go:356] "Generic (PLEG): container finished" podID="0b18da87-bf73-4749-9af5-9c8472c9a962" containerID="5cb5ea6920fbe133a7f74b17a0319ece9ea064edea4b69623b6cfd470ccc9263" exitCode=0 Sep 10 16:09:21.800437 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:21.800056 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4" event={"ID":"0b18da87-bf73-4749-9af5-9c8472c9a962","Type":"ContainerDied","Data":"5cb5ea6920fbe133a7f74b17a0319ece9ea064edea4b69623b6cfd470ccc9263"} Sep 10 16:09:21.800508 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:21.800493 2576 scope.go:117] "RemoveContainer" containerID="5cb5ea6920fbe133a7f74b17a0319ece9ea064edea4b69623b6cfd470ccc9263" Sep 10 16:09:21.801636 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:21.801609 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:09:21.801693 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:21.801680 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:09:21.801745 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:21.801717 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:09:22.812065 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:22.812018 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4" event={"ID":"0b18da87-bf73-4749-9af5-9c8472c9a962","Type":"ContainerStarted","Data":"5deb96806640ebf12f7ad2f64273f5ea3c18f2142253caa12f8411431c2bca43"} Sep 10 16:09:23.142082 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:23.141979 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4" Sep 10 16:09:23.142286 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:23.142256 2576 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Sep 10 16:09:23.142374 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:23.142322 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4" podUID="0b18da87-bf73-4749-9af5-9c8472c9a962" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Sep 10 16:09:24.142860 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:24.142820 2576 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Sep 10 16:09:24.143309 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:24.142880 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4" podUID="0b18da87-bf73-4749-9af5-9c8472c9a962" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Sep 10 16:09:25.142896 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:25.142855 2576 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Sep 10 16:09:25.143310 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:25.142916 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4" podUID="0b18da87-bf73-4749-9af5-9c8472c9a962" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Sep 10 16:09:25.576811 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:25.576773 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:09:26.142382 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:26.142345 2576 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Sep 10 16:09:26.142576 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:26.142421 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4" podUID="0b18da87-bf73-4749-9af5-9c8472c9a962" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Sep 10 16:09:26.233483 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:26.233447 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-044fdb-d7cvq"] Sep 10 16:09:26.237237 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:26.237217 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-044fdb-d7cvq" Sep 10 16:09:26.239623 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:26.239595 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 16:09:26.239788 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:26.239764 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwwht\"" Sep 10 16:09:26.246865 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:26.246822 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-044fdb-d7cvq"] Sep 10 16:09:26.352148 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:26.352111 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2cb5031-9bb2-40fb-838a-22322ce82da5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-044fdb-d7cvq\" (UID: \"d2cb5031-9bb2-40fb-838a-22322ce82da5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-044fdb-d7cvq" Sep 10 16:09:26.453951 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:26.453845 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2cb5031-9bb2-40fb-838a-22322ce82da5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-044fdb-d7cvq\" (UID: \"d2cb5031-9bb2-40fb-838a-22322ce82da5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-044fdb-d7cvq" Sep 10 16:09:26.454498 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:26.454476 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2cb5031-9bb2-40fb-838a-22322ce82da5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-044fdb-d7cvq\" (UID: \"d2cb5031-9bb2-40fb-838a-22322ce82da5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-044fdb-d7cvq" Sep 10 16:09:26.549100 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:26.549061 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-044fdb-d7cvq" Sep 10 16:09:26.856907 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:26.856827 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-044fdb-d7cvq"] Sep 10 16:09:26.859946 ip-10-0-139-236 kubenswrapper[2576]: W0910 16:09:26.859915 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2cb5031_9bb2_40fb_838a_22322ce82da5.slice/crio-6913f43329857d009ec97e1a1cee7b36d8ceca237f0bbaa6cd2556c96e87a291 WatchSource:0}: Error finding container 6913f43329857d009ec97e1a1cee7b36d8ceca237f0bbaa6cd2556c96e87a291: Status 404 returned error can't find the container with id 6913f43329857d009ec97e1a1cee7b36d8ceca237f0bbaa6cd2556c96e87a291 Sep 10 16:09:27.143208 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:27.143162 2576 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Sep 10 16:09:27.143399 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:27.143240 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4" podUID="0b18da87-bf73-4749-9af5-9c8472c9a962" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Sep 10 16:09:27.846915 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:27.846867 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-044fdb-d7cvq" event={"ID":"d2cb5031-9bb2-40fb-838a-22322ce82da5","Type":"ContainerStarted","Data":"aff8df95666b130cdc2cefbe0157f3ffa020ac5907e17619af89b761c281b556"} Sep 10 16:09:27.847347 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:27.846950 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-044fdb-d7cvq" event={"ID":"d2cb5031-9bb2-40fb-838a-22322ce82da5","Type":"ContainerStarted","Data":"6913f43329857d009ec97e1a1cee7b36d8ceca237f0bbaa6cd2556c96e87a291"} Sep 10 16:09:27.866011 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:27.865945 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-044fdb-d7cvq" podStartSLOduration=1.865923951 podStartE2EDuration="1.865923951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:09:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:09:27.864850566 +0000 UTC m=+1645.218369751" watchObservedRunningTime="2026-09-10 16:09:27.865923951 +0000 UTC m=+1645.219443119" Sep 10 16:09:28.146910 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:28.146823 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4" Sep 10 16:09:28.147126 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:28.147104 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4" Sep 10 16:09:28.147829 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:28.147808 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4" Sep 10 16:09:28.853556 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:28.853521 2576 generic.go:356] "Generic (PLEG): container finished" podID="d2cb5031-9bb2-40fb-838a-22322ce82da5" containerID="aff8df95666b130cdc2cefbe0157f3ffa020ac5907e17619af89b761c281b556" exitCode=0 Sep 10 16:09:28.854029 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:28.853612 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-044fdb-d7cvq" event={"ID":"d2cb5031-9bb2-40fb-838a-22322ce82da5","Type":"ContainerDied","Data":"aff8df95666b130cdc2cefbe0157f3ffa020ac5907e17619af89b761c281b556"} Sep 10 16:09:29.989551 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:29.989524 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-044fdb-d7cvq" Sep 10 16:09:30.087498 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:30.087456 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2cb5031-9bb2-40fb-838a-22322ce82da5-maas-api-service-ca\") pod \"d2cb5031-9bb2-40fb-838a-22322ce82da5\" (UID: \"d2cb5031-9bb2-40fb-838a-22322ce82da5\") " Sep 10 16:09:30.087880 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:30.087855 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2cb5031-9bb2-40fb-838a-22322ce82da5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d2cb5031-9bb2-40fb-838a-22322ce82da5" (UID: "d2cb5031-9bb2-40fb-838a-22322ce82da5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:09:30.189076 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:30.188993 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2cb5031-9bb2-40fb-838a-22322ce82da5-maas-api-service-ca\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:09:30.866487 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:30.866460 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-044fdb-d7cvq" Sep 10 16:09:30.866487 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:30.866469 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-044fdb-d7cvq" event={"ID":"d2cb5031-9bb2-40fb-838a-22322ce82da5","Type":"ContainerDied","Data":"6913f43329857d009ec97e1a1cee7b36d8ceca237f0bbaa6cd2556c96e87a291"} Sep 10 16:09:30.866803 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:30.866515 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6913f43329857d009ec97e1a1cee7b36d8ceca237f0bbaa6cd2556c96e87a291" Sep 10 16:09:32.621966 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:32.621925 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:09:32.662954 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:32.662925 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-044fdb-5898c975bc-cb7vn"] Sep 10 16:09:32.663222 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:32.663191 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-044fdb-5898c975bc-cb7vn" podUID="11736f45-7a39-4b92-b65f-2f7d3a2c5ed9" containerName="payload-processing" containerID="cri-o://6b71c0c75f1493b3d0ad7d8e22f9dc8d6081a454bf0667afc31b42755ac459eb" gracePeriod=30 Sep 10 16:09:32.678968 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:32.673179 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-044fdb-76bc9bcd5d-q2jnt"] Sep 10 16:09:32.678968 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:32.673474 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-044fdb-76bc9bcd5d-q2jnt" podUID="b230b042-b3ed-4599-994e-0f698ab9569b" containerName="payload-pre-processing" containerID="cri-o://dd52fb15084aa49f7ef4010d0eea8e95adf2a40defaf3cfc2ab84eb6ed75118f" gracePeriod=30 Sep 10 16:09:32.713134 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:32.713054 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:09:32.805611 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:32.805571 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:09:32.889194 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:32.888984 2576 generic.go:356] "Generic (PLEG): container finished" podID="11736f45-7a39-4b92-b65f-2f7d3a2c5ed9" containerID="6b71c0c75f1493b3d0ad7d8e22f9dc8d6081a454bf0667afc31b42755ac459eb" exitCode=0 Sep 10 16:09:32.889194 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:32.889087 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-044fdb-5898c975bc-cb7vn" event={"ID":"11736f45-7a39-4b92-b65f-2f7d3a2c5ed9","Type":"ContainerDied","Data":"6b71c0c75f1493b3d0ad7d8e22f9dc8d6081a454bf0667afc31b42755ac459eb"} Sep 10 16:09:32.892290 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:32.892130 2576 generic.go:356] "Generic (PLEG): container finished" podID="b230b042-b3ed-4599-994e-0f698ab9569b" containerID="dd52fb15084aa49f7ef4010d0eea8e95adf2a40defaf3cfc2ab84eb6ed75118f" exitCode=0 Sep 10 16:09:32.892290 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:32.892220 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-044fdb-76bc9bcd5d-q2jnt" event={"ID":"b230b042-b3ed-4599-994e-0f698ab9569b","Type":"ContainerDied","Data":"dd52fb15084aa49f7ef4010d0eea8e95adf2a40defaf3cfc2ab84eb6ed75118f"} Sep 10 16:09:32.913487 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:32.913444 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-044fdb-d7cvq"] Sep 10 16:09:32.914196 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:32.914164 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:09:32.917621 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:32.917587 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-044fdb-d7cvq"] Sep 10 16:09:32.954574 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:32.954547 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-044fdb-5898c975bc-cb7vn" Sep 10 16:09:32.981808 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:32.981782 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-044fdb-76bc9bcd5d-q2jnt" Sep 10 16:09:33.000906 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:33.000867 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:09:33.017143 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:33.017114 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gx44v\" (UniqueName: \"kubernetes.io/projected/11736f45-7a39-4b92-b65f-2f7d3a2c5ed9-kube-api-access-gx44v\") pod \"11736f45-7a39-4b92-b65f-2f7d3a2c5ed9\" (UID: \"11736f45-7a39-4b92-b65f-2f7d3a2c5ed9\") " Sep 10 16:09:33.017310 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:33.017235 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11736f45-7a39-4b92-b65f-2f7d3a2c5ed9-plugins-config-volume\") pod \"11736f45-7a39-4b92-b65f-2f7d3a2c5ed9\" (UID: \"11736f45-7a39-4b92-b65f-2f7d3a2c5ed9\") " Sep 10 16:09:33.017648 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:33.017628 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11736f45-7a39-4b92-b65f-2f7d3a2c5ed9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "11736f45-7a39-4b92-b65f-2f7d3a2c5ed9" (UID: "11736f45-7a39-4b92-b65f-2f7d3a2c5ed9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:09:33.019546 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:33.019509 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11736f45-7a39-4b92-b65f-2f7d3a2c5ed9-kube-api-access-gx44v" (OuterVolumeSpecName: "kube-api-access-gx44v") pod "11736f45-7a39-4b92-b65f-2f7d3a2c5ed9" (UID: "11736f45-7a39-4b92-b65f-2f7d3a2c5ed9"). InnerVolumeSpecName "kube-api-access-gx44v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:09:33.118633 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:33.118592 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v7xtw\" (UniqueName: \"kubernetes.io/projected/b230b042-b3ed-4599-994e-0f698ab9569b-kube-api-access-v7xtw\") pod \"b230b042-b3ed-4599-994e-0f698ab9569b\" (UID: \"b230b042-b3ed-4599-994e-0f698ab9569b\") " Sep 10 16:09:33.118816 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:33.118714 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b230b042-b3ed-4599-994e-0f698ab9569b-plugins-config-volume\") pod \"b230b042-b3ed-4599-994e-0f698ab9569b\" (UID: \"b230b042-b3ed-4599-994e-0f698ab9569b\") " Sep 10 16:09:33.119001 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:33.118984 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx44v\" (UniqueName: \"kubernetes.io/projected/11736f45-7a39-4b92-b65f-2f7d3a2c5ed9-kube-api-access-gx44v\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:09:33.119067 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:33.119003 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11736f45-7a39-4b92-b65f-2f7d3a2c5ed9-plugins-config-volume\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:09:33.119119 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:33.119057 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b230b042-b3ed-4599-994e-0f698ab9569b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b230b042-b3ed-4599-994e-0f698ab9569b" (UID: "b230b042-b3ed-4599-994e-0f698ab9569b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:09:33.120976 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:33.120952 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b230b042-b3ed-4599-994e-0f698ab9569b-kube-api-access-v7xtw" (OuterVolumeSpecName: "kube-api-access-v7xtw") pod "b230b042-b3ed-4599-994e-0f698ab9569b" (UID: "b230b042-b3ed-4599-994e-0f698ab9569b"). InnerVolumeSpecName "kube-api-access-v7xtw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:09:33.169961 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:33.169875 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2cb5031-9bb2-40fb-838a-22322ce82da5" path="/var/lib/kubelet/pods/d2cb5031-9bb2-40fb-838a-22322ce82da5/volumes" Sep 10 16:09:33.220039 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:33.220009 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7xtw\" (UniqueName: \"kubernetes.io/projected/b230b042-b3ed-4599-994e-0f698ab9569b-kube-api-access-v7xtw\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:09:33.220039 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:33.220036 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b230b042-b3ed-4599-994e-0f698ab9569b-plugins-config-volume\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:09:33.269631 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:33.269596 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:09:33.899169 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:33.899137 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-044fdb-76bc9bcd5d-q2jnt" Sep 10 16:09:33.899673 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:33.899162 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-044fdb-76bc9bcd5d-q2jnt" event={"ID":"b230b042-b3ed-4599-994e-0f698ab9569b","Type":"ContainerDied","Data":"ec887868fd0b68849cf0863bb91747ef6709fad124f99df1e11269e9d4625e47"} Sep 10 16:09:33.899673 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:33.899261 2576 scope.go:117] "RemoveContainer" containerID="dd52fb15084aa49f7ef4010d0eea8e95adf2a40defaf3cfc2ab84eb6ed75118f" Sep 10 16:09:33.900913 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:33.900881 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-044fdb-5898c975bc-cb7vn" Sep 10 16:09:33.901006 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:33.900874 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-044fdb-5898c975bc-cb7vn" event={"ID":"11736f45-7a39-4b92-b65f-2f7d3a2c5ed9","Type":"ContainerDied","Data":"a8772317b74ffa93058d5eb0f9d41a54aa70be7995913e1c90638fe025e67731"} Sep 10 16:09:33.911301 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:33.911281 2576 scope.go:117] "RemoveContainer" containerID="6b71c0c75f1493b3d0ad7d8e22f9dc8d6081a454bf0667afc31b42755ac459eb" Sep 10 16:09:33.923097 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:33.923070 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-044fdb-76bc9bcd5d-q2jnt"] Sep 10 16:09:33.927760 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:33.927716 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-044fdb-76bc9bcd5d-q2jnt"] Sep 10 16:09:33.943125 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:33.943050 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-044fdb-5898c975bc-cb7vn"] Sep 10 16:09:33.948235 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:33.948210 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-044fdb-5898c975bc-cb7vn"] Sep 10 16:09:35.169164 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:35.169127 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11736f45-7a39-4b92-b65f-2f7d3a2c5ed9" path="/var/lib/kubelet/pods/11736f45-7a39-4b92-b65f-2f7d3a2c5ed9/volumes" Sep 10 16:09:35.169607 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:35.169596 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b230b042-b3ed-4599-994e-0f698ab9569b" path="/var/lib/kubelet/pods/b230b042-b3ed-4599-994e-0f698ab9569b/volumes" Sep 10 16:09:37.191725 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:37.191686 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 16:09:39.677995 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:39.677953 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:09:39.784044 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:39.784003 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:09:39.924622 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:39.924583 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4"] Sep 10 16:09:39.924869 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:39.924847 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4" podUID="0b18da87-bf73-4749-9af5-9c8472c9a962" containerName="istio-proxy" containerID="cri-o://5deb96806640ebf12f7ad2f64273f5ea3c18f2142253caa12f8411431c2bca43" gracePeriod=30 Sep 10 16:09:39.940375 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:39.940294 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:09:43.149596 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:43.149561 2576 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 10 16:09:43.150119 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:43.149620 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4" podUID="0b18da87-bf73-4749-9af5-9c8472c9a962" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 16:09:43.725301 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:43.725263 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:09:45.189800 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:45.189701 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4" Sep 10 16:09:45.348905 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:45.348869 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b18da87-bf73-4749-9af5-9c8472c9a962-workload-socket\") pod \"0b18da87-bf73-4749-9af5-9c8472c9a962\" (UID: \"0b18da87-bf73-4749-9af5-9c8472c9a962\") " Sep 10 16:09:45.349117 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:45.348923 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b18da87-bf73-4749-9af5-9c8472c9a962-istio-envoy\") pod \"0b18da87-bf73-4749-9af5-9c8472c9a962\" (UID: \"0b18da87-bf73-4749-9af5-9c8472c9a962\") " Sep 10 16:09:45.349117 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:45.348951 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f59p8\" (UniqueName: \"kubernetes.io/projected/0b18da87-bf73-4749-9af5-9c8472c9a962-kube-api-access-f59p8\") pod \"0b18da87-bf73-4749-9af5-9c8472c9a962\" (UID: \"0b18da87-bf73-4749-9af5-9c8472c9a962\") " Sep 10 16:09:45.349117 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:45.348973 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b18da87-bf73-4749-9af5-9c8472c9a962-workload-certs\") pod \"0b18da87-bf73-4749-9af5-9c8472c9a962\" (UID: \"0b18da87-bf73-4749-9af5-9c8472c9a962\") " Sep 10 16:09:45.349117 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:45.348990 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b18da87-bf73-4749-9af5-9c8472c9a962-credential-socket\") pod \"0b18da87-bf73-4749-9af5-9c8472c9a962\" (UID: \"0b18da87-bf73-4749-9af5-9c8472c9a962\") " Sep 10 16:09:45.349117 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:45.349054 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b18da87-bf73-4749-9af5-9c8472c9a962-istio-data\") pod \"0b18da87-bf73-4749-9af5-9c8472c9a962\" (UID: \"0b18da87-bf73-4749-9af5-9c8472c9a962\") " Sep 10 16:09:45.349117 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:45.349104 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b18da87-bf73-4749-9af5-9c8472c9a962-istio-token\") pod \"0b18da87-bf73-4749-9af5-9c8472c9a962\" (UID: \"0b18da87-bf73-4749-9af5-9c8472c9a962\") " Sep 10 16:09:45.349506 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:45.349145 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b18da87-bf73-4749-9af5-9c8472c9a962-istiod-ca-cert\") pod \"0b18da87-bf73-4749-9af5-9c8472c9a962\" (UID: \"0b18da87-bf73-4749-9af5-9c8472c9a962\") " Sep 10 16:09:45.349506 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:45.349179 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b18da87-bf73-4749-9af5-9c8472c9a962-istio-podinfo\") pod \"0b18da87-bf73-4749-9af5-9c8472c9a962\" (UID: \"0b18da87-bf73-4749-9af5-9c8472c9a962\") " Sep 10 16:09:45.349506 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:45.349250 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b18da87-bf73-4749-9af5-9c8472c9a962-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0b18da87-bf73-4749-9af5-9c8472c9a962" (UID: "0b18da87-bf73-4749-9af5-9c8472c9a962"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:09:45.349506 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:45.349352 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b18da87-bf73-4749-9af5-9c8472c9a962-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0b18da87-bf73-4749-9af5-9c8472c9a962" (UID: "0b18da87-bf73-4749-9af5-9c8472c9a962"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:09:45.349713 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:45.349541 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b18da87-bf73-4749-9af5-9c8472c9a962-istio-data" (OuterVolumeSpecName: "istio-data") pod "0b18da87-bf73-4749-9af5-9c8472c9a962" (UID: "0b18da87-bf73-4749-9af5-9c8472c9a962"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:09:45.349713 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:45.349551 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b18da87-bf73-4749-9af5-9c8472c9a962-workload-socket\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:09:45.349713 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:45.349594 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b18da87-bf73-4749-9af5-9c8472c9a962-workload-certs\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:09:45.349713 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:45.349626 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b18da87-bf73-4749-9af5-9c8472c9a962-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0b18da87-bf73-4749-9af5-9c8472c9a962" (UID: "0b18da87-bf73-4749-9af5-9c8472c9a962"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:09:45.349713 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:45.349695 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b18da87-bf73-4749-9af5-9c8472c9a962-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0b18da87-bf73-4749-9af5-9c8472c9a962" (UID: "0b18da87-bf73-4749-9af5-9c8472c9a962"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:09:45.351597 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:45.351572 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b18da87-bf73-4749-9af5-9c8472c9a962-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0b18da87-bf73-4749-9af5-9c8472c9a962" (UID: "0b18da87-bf73-4749-9af5-9c8472c9a962"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 16:09:45.351770 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:45.351614 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b18da87-bf73-4749-9af5-9c8472c9a962-istio-token" (OuterVolumeSpecName: "istio-token") pod "0b18da87-bf73-4749-9af5-9c8472c9a962" (UID: "0b18da87-bf73-4749-9af5-9c8472c9a962"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:09:45.351770 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:45.351636 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b18da87-bf73-4749-9af5-9c8472c9a962-kube-api-access-f59p8" (OuterVolumeSpecName: "kube-api-access-f59p8") pod "0b18da87-bf73-4749-9af5-9c8472c9a962" (UID: "0b18da87-bf73-4749-9af5-9c8472c9a962"). InnerVolumeSpecName "kube-api-access-f59p8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:09:45.351770 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:45.351663 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b18da87-bf73-4749-9af5-9c8472c9a962-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0b18da87-bf73-4749-9af5-9c8472c9a962" (UID: "0b18da87-bf73-4749-9af5-9c8472c9a962"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:09:45.450949 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:45.450859 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b18da87-bf73-4749-9af5-9c8472c9a962-istio-envoy\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:09:45.450949 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:45.450891 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f59p8\" (UniqueName: \"kubernetes.io/projected/0b18da87-bf73-4749-9af5-9c8472c9a962-kube-api-access-f59p8\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:09:45.450949 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:45.450903 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b18da87-bf73-4749-9af5-9c8472c9a962-credential-socket\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:09:45.450949 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:45.450912 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b18da87-bf73-4749-9af5-9c8472c9a962-istio-data\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:09:45.450949 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:45.450920 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b18da87-bf73-4749-9af5-9c8472c9a962-istio-token\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:09:45.450949 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:45.450927 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b18da87-bf73-4749-9af5-9c8472c9a962-istiod-ca-cert\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:09:45.450949 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:45.450935 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b18da87-bf73-4749-9af5-9c8472c9a962-istio-podinfo\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:09:45.968415 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:45.968376 2576 generic.go:356] "Generic (PLEG): container finished" podID="0b18da87-bf73-4749-9af5-9c8472c9a962" containerID="5deb96806640ebf12f7ad2f64273f5ea3c18f2142253caa12f8411431c2bca43" exitCode=0 Sep 10 16:09:45.968621 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:45.968443 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4" Sep 10 16:09:45.968621 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:45.968458 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4" event={"ID":"0b18da87-bf73-4749-9af5-9c8472c9a962","Type":"ContainerDied","Data":"5deb96806640ebf12f7ad2f64273f5ea3c18f2142253caa12f8411431c2bca43"} Sep 10 16:09:45.968621 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:45.968540 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4" event={"ID":"0b18da87-bf73-4749-9af5-9c8472c9a962","Type":"ContainerDied","Data":"4f50c225de364bff00ab6bbc693c9eaf03fd4ebf886acc3b316eb6afe3fbdb3d"} Sep 10 16:09:45.968621 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:45.968564 2576 scope.go:117] "RemoveContainer" containerID="5deb96806640ebf12f7ad2f64273f5ea3c18f2142253caa12f8411431c2bca43" Sep 10 16:09:45.978899 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:45.978880 2576 scope.go:117] "RemoveContainer" containerID="5cb5ea6920fbe133a7f74b17a0319ece9ea064edea4b69623b6cfd470ccc9263" Sep 10 16:09:45.987962 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:45.987936 2576 scope.go:117] "RemoveContainer" containerID="5deb96806640ebf12f7ad2f64273f5ea3c18f2142253caa12f8411431c2bca43" Sep 10 16:09:45.988387 ip-10-0-139-236 kubenswrapper[2576]: E0910 16:09:45.988358 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5deb96806640ebf12f7ad2f64273f5ea3c18f2142253caa12f8411431c2bca43\": container with ID starting with 5deb96806640ebf12f7ad2f64273f5ea3c18f2142253caa12f8411431c2bca43 not found: ID does not exist" containerID="5deb96806640ebf12f7ad2f64273f5ea3c18f2142253caa12f8411431c2bca43" Sep 10 16:09:45.988549 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:45.988400 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5deb96806640ebf12f7ad2f64273f5ea3c18f2142253caa12f8411431c2bca43"} err="failed to get container status \"5deb96806640ebf12f7ad2f64273f5ea3c18f2142253caa12f8411431c2bca43\": rpc error: code = NotFound desc = could not find container \"5deb96806640ebf12f7ad2f64273f5ea3c18f2142253caa12f8411431c2bca43\": container with ID starting with 5deb96806640ebf12f7ad2f64273f5ea3c18f2142253caa12f8411431c2bca43 not found: ID does not exist" Sep 10 16:09:45.988549 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:45.988426 2576 scope.go:117] "RemoveContainer" containerID="5cb5ea6920fbe133a7f74b17a0319ece9ea064edea4b69623b6cfd470ccc9263" Sep 10 16:09:45.988818 ip-10-0-139-236 kubenswrapper[2576]: E0910 16:09:45.988792 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cb5ea6920fbe133a7f74b17a0319ece9ea064edea4b69623b6cfd470ccc9263\": container with ID starting with 5cb5ea6920fbe133a7f74b17a0319ece9ea064edea4b69623b6cfd470ccc9263 not found: ID does not exist" containerID="5cb5ea6920fbe133a7f74b17a0319ece9ea064edea4b69623b6cfd470ccc9263" Sep 10 16:09:45.988907 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:45.988828 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cb5ea6920fbe133a7f74b17a0319ece9ea064edea4b69623b6cfd470ccc9263"} err="failed to get container status \"5cb5ea6920fbe133a7f74b17a0319ece9ea064edea4b69623b6cfd470ccc9263\": rpc error: code = NotFound desc = could not find container \"5cb5ea6920fbe133a7f74b17a0319ece9ea064edea4b69623b6cfd470ccc9263\": container with ID starting with 5cb5ea6920fbe133a7f74b17a0319ece9ea064edea4b69623b6cfd470ccc9263 not found: ID does not exist" Sep 10 16:09:45.990922 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:45.990899 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4"] Sep 10 16:09:45.994681 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:45.994656 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-044fdb-openshift-default-6896cd568d-h5wh4"] Sep 10 16:09:47.169869 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:47.169837 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b18da87-bf73-4749-9af5-9c8472c9a962" path="/var/lib/kubelet/pods/0b18da87-bf73-4749-9af5-9c8472c9a962/volumes" Sep 10 16:09:56.892354 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:56.892322 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 16:09:59.219363 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:59.219329 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:09:59.249633 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:59.249598 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c0ec41-69664b596f-642w8"] Sep 10 16:09:59.249878 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:59.249848 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-c0ec41-69664b596f-642w8" podUID="168d3721-ce5b-42b7-b3c5-44106c77ac3f" containerName="payload-processing" containerID="cri-o://a5df31be123029b05f86acd3e73374c2a1e2602d238aeb08247900f6e7277787" gracePeriod=30 Sep 10 16:09:59.527149 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:59.527114 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c0ec41-69664b596f-642w8" Sep 10 16:09:59.586709 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:59.586672 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/168d3721-ce5b-42b7-b3c5-44106c77ac3f-plugins-config-volume\") pod \"168d3721-ce5b-42b7-b3c5-44106c77ac3f\" (UID: \"168d3721-ce5b-42b7-b3c5-44106c77ac3f\") " Sep 10 16:09:59.586940 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:59.586768 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r8n5g\" (UniqueName: \"kubernetes.io/projected/168d3721-ce5b-42b7-b3c5-44106c77ac3f-kube-api-access-r8n5g\") pod \"168d3721-ce5b-42b7-b3c5-44106c77ac3f\" (UID: \"168d3721-ce5b-42b7-b3c5-44106c77ac3f\") " Sep 10 16:09:59.587152 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:59.587125 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/168d3721-ce5b-42b7-b3c5-44106c77ac3f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "168d3721-ce5b-42b7-b3c5-44106c77ac3f" (UID: "168d3721-ce5b-42b7-b3c5-44106c77ac3f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:09:59.589118 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:59.589098 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/168d3721-ce5b-42b7-b3c5-44106c77ac3f-kube-api-access-r8n5g" (OuterVolumeSpecName: "kube-api-access-r8n5g") pod "168d3721-ce5b-42b7-b3c5-44106c77ac3f" (UID: "168d3721-ce5b-42b7-b3c5-44106c77ac3f"). InnerVolumeSpecName "kube-api-access-r8n5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:09:59.687771 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:59.687721 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/168d3721-ce5b-42b7-b3c5-44106c77ac3f-plugins-config-volume\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:09:59.687771 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:09:59.687766 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8n5g\" (UniqueName: \"kubernetes.io/projected/168d3721-ce5b-42b7-b3c5-44106c77ac3f-kube-api-access-r8n5g\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:10:00.045709 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:10:00.045672 2576 generic.go:356] "Generic (PLEG): container finished" podID="168d3721-ce5b-42b7-b3c5-44106c77ac3f" containerID="a5df31be123029b05f86acd3e73374c2a1e2602d238aeb08247900f6e7277787" exitCode=0 Sep 10 16:10:00.045904 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:10:00.045715 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c0ec41-69664b596f-642w8" Sep 10 16:10:00.045904 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:10:00.045764 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c0ec41-69664b596f-642w8" event={"ID":"168d3721-ce5b-42b7-b3c5-44106c77ac3f","Type":"ContainerDied","Data":"a5df31be123029b05f86acd3e73374c2a1e2602d238aeb08247900f6e7277787"} Sep 10 16:10:00.045904 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:10:00.045821 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c0ec41-69664b596f-642w8" event={"ID":"168d3721-ce5b-42b7-b3c5-44106c77ac3f","Type":"ContainerDied","Data":"67b45f2aa2869fbe81b5749b74af2816fdb74027b7b4e006f76332a02546f540"} Sep 10 16:10:00.045904 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:10:00.045846 2576 scope.go:117] "RemoveContainer" containerID="a5df31be123029b05f86acd3e73374c2a1e2602d238aeb08247900f6e7277787" Sep 10 16:10:00.055505 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:10:00.055485 2576 scope.go:117] "RemoveContainer" containerID="a5df31be123029b05f86acd3e73374c2a1e2602d238aeb08247900f6e7277787" Sep 10 16:10:00.055827 ip-10-0-139-236 kubenswrapper[2576]: E0910 16:10:00.055808 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5df31be123029b05f86acd3e73374c2a1e2602d238aeb08247900f6e7277787\": container with ID starting with a5df31be123029b05f86acd3e73374c2a1e2602d238aeb08247900f6e7277787 not found: ID does not exist" containerID="a5df31be123029b05f86acd3e73374c2a1e2602d238aeb08247900f6e7277787" Sep 10 16:10:00.055907 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:10:00.055838 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5df31be123029b05f86acd3e73374c2a1e2602d238aeb08247900f6e7277787"} err="failed to get container status \"a5df31be123029b05f86acd3e73374c2a1e2602d238aeb08247900f6e7277787\": rpc error: code = NotFound desc = could not find container \"a5df31be123029b05f86acd3e73374c2a1e2602d238aeb08247900f6e7277787\": container with ID starting with a5df31be123029b05f86acd3e73374c2a1e2602d238aeb08247900f6e7277787 not found: ID does not exist" Sep 10 16:10:00.069588 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:10:00.069554 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c0ec41-69664b596f-642w8"] Sep 10 16:10:00.072172 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:10:00.072149 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c0ec41-69664b596f-642w8"] Sep 10 16:10:00.931101 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:10:00.931035 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:10:01.170010 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:10:01.169976 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="168d3721-ce5b-42b7-b3c5-44106c77ac3f" path="/var/lib/kubelet/pods/168d3721-ce5b-42b7-b3c5-44106c77ac3f/volumes" Sep 10 16:10:03.485653 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:10:03.485618 2576 scope.go:117] "RemoveContainer" containerID="59416fe0c08ffe67d661e47af765205d04e19a2c9467c2afa3ebbc101ac08ee8" Sep 10 16:10:05.185031 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:10:05.184971 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:10:10.285895 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:10:10.285849 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 16:10:24.568610 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:10:24.568495 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:10:41.456861 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:10:41.456816 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:10:43.884346 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:10:43.884303 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:10:49.579283 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:10:49.579236 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:10:49.657284 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:10:49.657246 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:10:49.925238 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:10:49.925141 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:10:50.238974 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:10:50.238881 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e9f330-n4kh9"] Sep 10 16:10:50.239535 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:10:50.239506 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b18da87-bf73-4749-9af5-9c8472c9a962" containerName="istio-proxy" Sep 10 16:10:50.239535 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:10:50.239534 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b18da87-bf73-4749-9af5-9c8472c9a962" containerName="istio-proxy" Sep 10 16:10:50.239721 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:10:50.239569 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b18da87-bf73-4749-9af5-9c8472c9a962" containerName="istio-proxy" Sep 10 16:10:50.239721 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:10:50.239578 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b18da87-bf73-4749-9af5-9c8472c9a962" containerName="istio-proxy" Sep 10 16:10:50.239721 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:10:50.239587 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b230b042-b3ed-4599-994e-0f698ab9569b" containerName="payload-pre-processing" Sep 10 16:10:50.239721 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:10:50.239598 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b230b042-b3ed-4599-994e-0f698ab9569b" containerName="payload-pre-processing" Sep 10 16:10:50.239721 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:10:50.239612 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11736f45-7a39-4b92-b65f-2f7d3a2c5ed9" containerName="payload-processing" Sep 10 16:10:50.239721 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:10:50.239621 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="11736f45-7a39-4b92-b65f-2f7d3a2c5ed9" containerName="payload-processing" Sep 10 16:10:50.239721 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:10:50.239634 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2cb5031-9bb2-40fb-838a-22322ce82da5" containerName="revoke-keys" Sep 10 16:10:50.239721 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:10:50.239642 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2cb5031-9bb2-40fb-838a-22322ce82da5" containerName="revoke-keys" Sep 10 16:10:50.239721 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:10:50.239653 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="168d3721-ce5b-42b7-b3c5-44106c77ac3f" containerName="payload-processing" Sep 10 16:10:50.239721 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:10:50.239660 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="168d3721-ce5b-42b7-b3c5-44106c77ac3f" containerName="payload-processing" Sep 10 16:10:50.240153 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:10:50.239792 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b18da87-bf73-4749-9af5-9c8472c9a962" containerName="istio-proxy" Sep 10 16:10:50.240153 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:10:50.239804 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2cb5031-9bb2-40fb-838a-22322ce82da5" containerName="revoke-keys" Sep 10 16:10:50.240153 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:10:50.239813 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b18da87-bf73-4749-9af5-9c8472c9a962" containerName="istio-proxy" Sep 10 16:10:50.240153 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:10:50.239823 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b230b042-b3ed-4599-994e-0f698ab9569b" containerName="payload-pre-processing" Sep 10 16:10:50.240153 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:10:50.239837 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="168d3721-ce5b-42b7-b3c5-44106c77ac3f" containerName="payload-processing" Sep 10 16:10:50.240153 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:10:50.239846 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="11736f45-7a39-4b92-b65f-2f7d3a2c5ed9" containerName="payload-processing" Sep 10 16:10:50.243213 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:10:50.243193 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e9f330-n4kh9" Sep 10 16:10:50.246011 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:10:50.245989 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 16:10:50.246135 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:10:50.245989 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwwht\"" Sep 10 16:10:50.253454 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:10:50.253424 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e9f330-n4kh9"] Sep 10 16:10:50.391890 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:10:50.391838 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eacc589c-de4a-4176-8847-616bc6af0834-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e9f330-n4kh9\" (UID: \"eacc589c-de4a-4176-8847-616bc6af0834\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e9f330-n4kh9" Sep 10 16:10:50.465987 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:10:50.465947 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:10:50.493188 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:10:50.493087 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eacc589c-de4a-4176-8847-616bc6af0834-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e9f330-n4kh9\" (UID: \"eacc589c-de4a-4176-8847-616bc6af0834\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e9f330-n4kh9" Sep 10 16:10:50.493697 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:10:50.493677 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eacc589c-de4a-4176-8847-616bc6af0834-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e9f330-n4kh9\" (UID: \"eacc589c-de4a-4176-8847-616bc6af0834\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e9f330-n4kh9" Sep 10 16:10:50.555325 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:10:50.555287 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e9f330-n4kh9" Sep 10 16:10:50.831863 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:10:50.831835 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e9f330-n4kh9"] Sep 10 16:10:50.833719 ip-10-0-139-236 kubenswrapper[2576]: W0910 16:10:50.833685 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeacc589c_de4a_4176_8847_616bc6af0834.slice/crio-86c2cab66f0c893e63818c916a80ce430d32cd8816222d7639e94b286507e212 WatchSource:0}: Error finding container 86c2cab66f0c893e63818c916a80ce430d32cd8816222d7639e94b286507e212: Status 404 returned error can't find the container with id 86c2cab66f0c893e63818c916a80ce430d32cd8816222d7639e94b286507e212 Sep 10 16:10:51.313922 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:10:51.313880 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e9f330-n4kh9" event={"ID":"eacc589c-de4a-4176-8847-616bc6af0834","Type":"ContainerStarted","Data":"1118831b6d3af2fb3e42cea039dff78260eadf60b43bed102ff47aa33ba987e3"} Sep 10 16:10:51.314116 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:10:51.313942 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e9f330-n4kh9" event={"ID":"eacc589c-de4a-4176-8847-616bc6af0834","Type":"ContainerStarted","Data":"86c2cab66f0c893e63818c916a80ce430d32cd8816222d7639e94b286507e212"} Sep 10 16:10:51.331189 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:10:51.331136 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e9f330-n4kh9" podStartSLOduration=1.331118947 podStartE2EDuration="1.331118947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:10:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:10:51.32927736 +0000 UTC m=+1728.682796524" watchObservedRunningTime="2026-09-10 16:10:51.331118947 +0000 UTC m=+1728.684638110" Sep 10 16:10:52.320198 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:10:52.320160 2576 generic.go:356] "Generic (PLEG): container finished" podID="eacc589c-de4a-4176-8847-616bc6af0834" containerID="1118831b6d3af2fb3e42cea039dff78260eadf60b43bed102ff47aa33ba987e3" exitCode=0 Sep 10 16:10:52.320622 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:10:52.320250 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e9f330-n4kh9" event={"ID":"eacc589c-de4a-4176-8847-616bc6af0834","Type":"ContainerDied","Data":"1118831b6d3af2fb3e42cea039dff78260eadf60b43bed102ff47aa33ba987e3"} Sep 10 16:10:52.847375 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:10:52.847321 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:10:53.468218 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:10:53.468152 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e9f330-n4kh9" Sep 10 16:10:53.626325 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:10:53.625536 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eacc589c-de4a-4176-8847-616bc6af0834-maas-api-service-ca\") pod \"eacc589c-de4a-4176-8847-616bc6af0834\" (UID: \"eacc589c-de4a-4176-8847-616bc6af0834\") " Sep 10 16:10:53.627831 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:10:53.626697 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eacc589c-de4a-4176-8847-616bc6af0834-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "eacc589c-de4a-4176-8847-616bc6af0834" (UID: "eacc589c-de4a-4176-8847-616bc6af0834"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:10:53.727143 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:10:53.727048 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eacc589c-de4a-4176-8847-616bc6af0834-maas-api-service-ca\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:10:54.331787 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:10:54.331721 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e9f330-n4kh9" Sep 10 16:10:54.331787 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:10:54.331772 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e9f330-n4kh9" event={"ID":"eacc589c-de4a-4176-8847-616bc6af0834","Type":"ContainerDied","Data":"86c2cab66f0c893e63818c916a80ce430d32cd8816222d7639e94b286507e212"} Sep 10 16:10:54.332064 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:10:54.331831 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86c2cab66f0c893e63818c916a80ce430d32cd8816222d7639e94b286507e212" Sep 10 16:11:00.534607 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:00.534558 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:11:01.128409 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:01.128369 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:11:01.160918 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:01.160881 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e9f330-5d469f898b-tz9jl"] Sep 10 16:11:01.161196 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:01.161167 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e9f330-5d469f898b-tz9jl" podUID="0d80d4de-29ec-4b05-899d-7f9c9e6778be" containerName="payload-pre-processing" containerID="cri-o://0d9c717825a20d3adbb84467c2483b5aac050f7db6199524c44dd90e9f04b23e" gracePeriod=30 Sep 10 16:11:01.231614 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:01.231563 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:11:01.317423 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:01.317386 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e9f330-n4kh9"] Sep 10 16:11:01.323703 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:01.323665 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e9f330-n4kh9"] Sep 10 16:11:01.323981 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:01.323939 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:11:01.369694 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:01.369664 2576 generic.go:356] "Generic (PLEG): container finished" podID="0d80d4de-29ec-4b05-899d-7f9c9e6778be" containerID="0d9c717825a20d3adbb84467c2483b5aac050f7db6199524c44dd90e9f04b23e" exitCode=0 Sep 10 16:11:01.369932 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:01.369766 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e9f330-5d469f898b-tz9jl" event={"ID":"0d80d4de-29ec-4b05-899d-7f9c9e6778be","Type":"ContainerDied","Data":"0d9c717825a20d3adbb84467c2483b5aac050f7db6199524c44dd90e9f04b23e"} Sep 10 16:11:01.421816 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:01.421479 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:11:01.450135 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:01.450101 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e9f330-5d469f898b-tz9jl" Sep 10 16:11:01.605310 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:01.605274 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkpbb\" (UniqueName: \"kubernetes.io/projected/0d80d4de-29ec-4b05-899d-7f9c9e6778be-kube-api-access-jkpbb\") pod \"0d80d4de-29ec-4b05-899d-7f9c9e6778be\" (UID: \"0d80d4de-29ec-4b05-899d-7f9c9e6778be\") " Sep 10 16:11:01.605752 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:01.605349 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d80d4de-29ec-4b05-899d-7f9c9e6778be-plugins-config-volume\") pod \"0d80d4de-29ec-4b05-899d-7f9c9e6778be\" (UID: \"0d80d4de-29ec-4b05-899d-7f9c9e6778be\") " Sep 10 16:11:01.605819 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:01.605723 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d80d4de-29ec-4b05-899d-7f9c9e6778be-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0d80d4de-29ec-4b05-899d-7f9c9e6778be" (UID: "0d80d4de-29ec-4b05-899d-7f9c9e6778be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:11:01.607549 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:01.607528 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d80d4de-29ec-4b05-899d-7f9c9e6778be-kube-api-access-jkpbb" (OuterVolumeSpecName: "kube-api-access-jkpbb") pod "0d80d4de-29ec-4b05-899d-7f9c9e6778be" (UID: "0d80d4de-29ec-4b05-899d-7f9c9e6778be"). InnerVolumeSpecName "kube-api-access-jkpbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:11:01.707347 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:01.707259 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkpbb\" (UniqueName: \"kubernetes.io/projected/0d80d4de-29ec-4b05-899d-7f9c9e6778be-kube-api-access-jkpbb\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:11:01.707347 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:01.707293 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d80d4de-29ec-4b05-899d-7f9c9e6778be-plugins-config-volume\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:11:01.902557 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:01.902515 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:11:02.376933 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:02.376895 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e9f330-5d469f898b-tz9jl" event={"ID":"0d80d4de-29ec-4b05-899d-7f9c9e6778be","Type":"ContainerDied","Data":"44a16486fbfc2c8dcb6c89bee843137389cfed32ff8f19920aa61e4c22998bd7"} Sep 10 16:11:02.376933 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:02.376930 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e9f330-5d469f898b-tz9jl" Sep 10 16:11:02.377178 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:02.376962 2576 scope.go:117] "RemoveContainer" containerID="0d9c717825a20d3adbb84467c2483b5aac050f7db6199524c44dd90e9f04b23e" Sep 10 16:11:02.399715 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:02.399682 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e9f330-5d469f898b-tz9jl"] Sep 10 16:11:02.402988 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:02.402961 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e9f330-5d469f898b-tz9jl"] Sep 10 16:11:03.169791 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:03.169752 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d80d4de-29ec-4b05-899d-7f9c9e6778be" path="/var/lib/kubelet/pods/0d80d4de-29ec-4b05-899d-7f9c9e6778be/volumes" Sep 10 16:11:03.170182 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:03.170113 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eacc589c-de4a-4176-8847-616bc6af0834" path="/var/lib/kubelet/pods/eacc589c-de4a-4176-8847-616bc6af0834/volumes" Sep 10 16:11:04.387389 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:04.387353 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 16:11:07.709670 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:07.709633 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:11:07.810533 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:07.810495 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:11:07.967998 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:07.967901 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc"] Sep 10 16:11:07.968197 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:07.968130 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc" podUID="19052e67-3c56-4052-b5be-05e93c3cdd69" containerName="istio-proxy" containerID="cri-o://2dc3f7d130f1f360127d4453c7809c0a98c5c5996ea93318c153a3bc3c12515f" gracePeriod=30 Sep 10 16:11:08.005762 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:08.005699 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:11:08.537282 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:08.537244 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:11:13.229037 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:13.228964 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc" Sep 10 16:11:13.318267 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:13.318228 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19052e67-3c56-4052-b5be-05e93c3cdd69-istio-envoy\") pod \"19052e67-3c56-4052-b5be-05e93c3cdd69\" (UID: \"19052e67-3c56-4052-b5be-05e93c3cdd69\") " Sep 10 16:11:13.318476 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:13.318295 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19052e67-3c56-4052-b5be-05e93c3cdd69-istiod-ca-cert\") pod \"19052e67-3c56-4052-b5be-05e93c3cdd69\" (UID: \"19052e67-3c56-4052-b5be-05e93c3cdd69\") " Sep 10 16:11:13.318476 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:13.318331 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19052e67-3c56-4052-b5be-05e93c3cdd69-istio-token\") pod \"19052e67-3c56-4052-b5be-05e93c3cdd69\" (UID: \"19052e67-3c56-4052-b5be-05e93c3cdd69\") " Sep 10 16:11:13.318476 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:13.318375 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tpf4x\" (UniqueName: \"kubernetes.io/projected/19052e67-3c56-4052-b5be-05e93c3cdd69-kube-api-access-tpf4x\") pod \"19052e67-3c56-4052-b5be-05e93c3cdd69\" (UID: \"19052e67-3c56-4052-b5be-05e93c3cdd69\") " Sep 10 16:11:13.318665 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:13.318500 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19052e67-3c56-4052-b5be-05e93c3cdd69-credential-socket\") pod \"19052e67-3c56-4052-b5be-05e93c3cdd69\" (UID: \"19052e67-3c56-4052-b5be-05e93c3cdd69\") " Sep 10 16:11:13.318665 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:13.318561 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19052e67-3c56-4052-b5be-05e93c3cdd69-istio-podinfo\") pod \"19052e67-3c56-4052-b5be-05e93c3cdd69\" (UID: \"19052e67-3c56-4052-b5be-05e93c3cdd69\") " Sep 10 16:11:13.318665 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:13.318594 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19052e67-3c56-4052-b5be-05e93c3cdd69-istio-data\") pod \"19052e67-3c56-4052-b5be-05e93c3cdd69\" (UID: \"19052e67-3c56-4052-b5be-05e93c3cdd69\") " Sep 10 16:11:13.318665 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:13.318630 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19052e67-3c56-4052-b5be-05e93c3cdd69-workload-socket\") pod \"19052e67-3c56-4052-b5be-05e93c3cdd69\" (UID: \"19052e67-3c56-4052-b5be-05e93c3cdd69\") " Sep 10 16:11:13.318898 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:13.318678 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19052e67-3c56-4052-b5be-05e93c3cdd69-workload-certs\") pod \"19052e67-3c56-4052-b5be-05e93c3cdd69\" (UID: \"19052e67-3c56-4052-b5be-05e93c3cdd69\") " Sep 10 16:11:13.318898 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:13.318805 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19052e67-3c56-4052-b5be-05e93c3cdd69-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "19052e67-3c56-4052-b5be-05e93c3cdd69" (UID: "19052e67-3c56-4052-b5be-05e93c3cdd69"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:11:13.319002 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:13.318907 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19052e67-3c56-4052-b5be-05e93c3cdd69-istio-data" (OuterVolumeSpecName: "istio-data") pod "19052e67-3c56-4052-b5be-05e93c3cdd69" (UID: "19052e67-3c56-4052-b5be-05e93c3cdd69"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:11:13.319130 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:13.319110 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19052e67-3c56-4052-b5be-05e93c3cdd69-istio-data\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:11:13.319203 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:13.319139 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19052e67-3c56-4052-b5be-05e93c3cdd69-istiod-ca-cert\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:11:13.319203 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:13.319116 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19052e67-3c56-4052-b5be-05e93c3cdd69-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "19052e67-3c56-4052-b5be-05e93c3cdd69" (UID: "19052e67-3c56-4052-b5be-05e93c3cdd69"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:11:13.319203 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:13.319128 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19052e67-3c56-4052-b5be-05e93c3cdd69-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "19052e67-3c56-4052-b5be-05e93c3cdd69" (UID: "19052e67-3c56-4052-b5be-05e93c3cdd69"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:11:13.319312 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:13.319246 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19052e67-3c56-4052-b5be-05e93c3cdd69-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "19052e67-3c56-4052-b5be-05e93c3cdd69" (UID: "19052e67-3c56-4052-b5be-05e93c3cdd69"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:11:13.321414 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:13.321387 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19052e67-3c56-4052-b5be-05e93c3cdd69-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "19052e67-3c56-4052-b5be-05e93c3cdd69" (UID: "19052e67-3c56-4052-b5be-05e93c3cdd69"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:11:13.321483 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:13.321406 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19052e67-3c56-4052-b5be-05e93c3cdd69-kube-api-access-tpf4x" (OuterVolumeSpecName: "kube-api-access-tpf4x") pod "19052e67-3c56-4052-b5be-05e93c3cdd69" (UID: "19052e67-3c56-4052-b5be-05e93c3cdd69"). InnerVolumeSpecName "kube-api-access-tpf4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:11:13.321483 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:13.321433 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19052e67-3c56-4052-b5be-05e93c3cdd69-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "19052e67-3c56-4052-b5be-05e93c3cdd69" (UID: "19052e67-3c56-4052-b5be-05e93c3cdd69"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 16:11:13.321483 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:13.321437 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19052e67-3c56-4052-b5be-05e93c3cdd69-istio-token" (OuterVolumeSpecName: "istio-token") pod "19052e67-3c56-4052-b5be-05e93c3cdd69" (UID: "19052e67-3c56-4052-b5be-05e93c3cdd69"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:11:13.419906 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:13.419847 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19052e67-3c56-4052-b5be-05e93c3cdd69-istio-envoy\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:11:13.419906 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:13.419895 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19052e67-3c56-4052-b5be-05e93c3cdd69-istio-token\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:11:13.419906 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:13.419910 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpf4x\" (UniqueName: \"kubernetes.io/projected/19052e67-3c56-4052-b5be-05e93c3cdd69-kube-api-access-tpf4x\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:11:13.419906 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:13.419923 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19052e67-3c56-4052-b5be-05e93c3cdd69-credential-socket\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:11:13.420210 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:13.419935 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19052e67-3c56-4052-b5be-05e93c3cdd69-istio-podinfo\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:11:13.420210 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:13.419947 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19052e67-3c56-4052-b5be-05e93c3cdd69-workload-socket\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:11:13.420210 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:13.419957 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19052e67-3c56-4052-b5be-05e93c3cdd69-workload-certs\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:11:13.436342 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:13.436309 2576 generic.go:356] "Generic (PLEG): container finished" podID="19052e67-3c56-4052-b5be-05e93c3cdd69" containerID="2dc3f7d130f1f360127d4453c7809c0a98c5c5996ea93318c153a3bc3c12515f" exitCode=0 Sep 10 16:11:13.436519 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:13.436360 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc" Sep 10 16:11:13.436519 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:13.436359 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc" event={"ID":"19052e67-3c56-4052-b5be-05e93c3cdd69","Type":"ContainerDied","Data":"2dc3f7d130f1f360127d4453c7809c0a98c5c5996ea93318c153a3bc3c12515f"} Sep 10 16:11:13.436519 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:13.436486 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc" event={"ID":"19052e67-3c56-4052-b5be-05e93c3cdd69","Type":"ContainerDied","Data":"c46ce460eeebb4b5606901301ce2157ad8ddb483fffea18c5b5cdecb843177e5"} Sep 10 16:11:13.436519 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:13.436517 2576 scope.go:117] "RemoveContainer" containerID="2dc3f7d130f1f360127d4453c7809c0a98c5c5996ea93318c153a3bc3c12515f" Sep 10 16:11:13.447198 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:13.447173 2576 scope.go:117] "RemoveContainer" containerID="61edc4c4d96f3d0ffea786997c6a8857b67bd72d0347664d18adc659d663a0cd" Sep 10 16:11:13.456262 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:13.456237 2576 scope.go:117] "RemoveContainer" containerID="2dc3f7d130f1f360127d4453c7809c0a98c5c5996ea93318c153a3bc3c12515f" Sep 10 16:11:13.456618 ip-10-0-139-236 kubenswrapper[2576]: E0910 16:11:13.456594 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dc3f7d130f1f360127d4453c7809c0a98c5c5996ea93318c153a3bc3c12515f\": container with ID starting with 2dc3f7d130f1f360127d4453c7809c0a98c5c5996ea93318c153a3bc3c12515f not found: ID does not exist" containerID="2dc3f7d130f1f360127d4453c7809c0a98c5c5996ea93318c153a3bc3c12515f" Sep 10 16:11:13.456703 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:13.456633 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dc3f7d130f1f360127d4453c7809c0a98c5c5996ea93318c153a3bc3c12515f"} err="failed to get container status \"2dc3f7d130f1f360127d4453c7809c0a98c5c5996ea93318c153a3bc3c12515f\": rpc error: code = NotFound desc = could not find container \"2dc3f7d130f1f360127d4453c7809c0a98c5c5996ea93318c153a3bc3c12515f\": container with ID starting with 2dc3f7d130f1f360127d4453c7809c0a98c5c5996ea93318c153a3bc3c12515f not found: ID does not exist" Sep 10 16:11:13.456703 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:13.456660 2576 scope.go:117] "RemoveContainer" containerID="61edc4c4d96f3d0ffea786997c6a8857b67bd72d0347664d18adc659d663a0cd" Sep 10 16:11:13.457006 ip-10-0-139-236 kubenswrapper[2576]: E0910 16:11:13.456987 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61edc4c4d96f3d0ffea786997c6a8857b67bd72d0347664d18adc659d663a0cd\": container with ID starting with 61edc4c4d96f3d0ffea786997c6a8857b67bd72d0347664d18adc659d663a0cd not found: ID does not exist" containerID="61edc4c4d96f3d0ffea786997c6a8857b67bd72d0347664d18adc659d663a0cd" Sep 10 16:11:13.457053 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:13.457014 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61edc4c4d96f3d0ffea786997c6a8857b67bd72d0347664d18adc659d663a0cd"} err="failed to get container status \"61edc4c4d96f3d0ffea786997c6a8857b67bd72d0347664d18adc659d663a0cd\": rpc error: code = NotFound desc = could not find container \"61edc4c4d96f3d0ffea786997c6a8857b67bd72d0347664d18adc659d663a0cd\": container with ID starting with 61edc4c4d96f3d0ffea786997c6a8857b67bd72d0347664d18adc659d663a0cd not found: ID does not exist" Sep 10 16:11:13.462950 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:13.462918 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc"] Sep 10 16:11:13.465323 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:13.465298 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-e9f330-openshift-default-5d77fbb964-db5jc"] Sep 10 16:11:15.169150 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:15.169106 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19052e67-3c56-4052-b5be-05e93c3cdd69" path="/var/lib/kubelet/pods/19052e67-3c56-4052-b5be-05e93c3cdd69/volumes" Sep 10 16:11:18.939107 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:18.939051 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:11:18.973187 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:18.973151 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-50ed73-7c8bb55487-kpbvm"] Sep 10 16:11:18.973466 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:18.973431 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-50ed73-7c8bb55487-kpbvm" podUID="ecfb8c74-1806-43d3-8d8d-573617251bb9" containerName="payload-processing" containerID="cri-o://bebe906e4d8dcc38a67a9310429fce2f20de4d17b5e98f9434fbdad6a7a9f573" gracePeriod=30 Sep 10 16:11:18.992330 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:18.992293 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-50ed73-657b6f9cdb-4xh7x"] Sep 10 16:11:18.992561 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:18.992531 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-50ed73-657b6f9cdb-4xh7x" podUID="c6c987d0-fdd8-4180-9542-1020e83a704d" containerName="payload-pre-processing" containerID="cri-o://de2dfe232787b7b0788756e1a9e5aa9b0fd67ed6509bfee7057fb71a48ec250b" gracePeriod=30 Sep 10 16:11:19.048240 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:19.048208 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:11:19.147123 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:19.147054 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:11:19.228935 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:19.228888 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:11:19.263784 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:19.263754 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-50ed73-7c8bb55487-kpbvm" Sep 10 16:11:19.303935 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:19.303900 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-50ed73-657b6f9cdb-4xh7x" Sep 10 16:11:19.311488 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:19.311437 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:11:19.380929 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:19.380889 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fz4cw\" (UniqueName: \"kubernetes.io/projected/c6c987d0-fdd8-4180-9542-1020e83a704d-kube-api-access-fz4cw\") pod \"c6c987d0-fdd8-4180-9542-1020e83a704d\" (UID: \"c6c987d0-fdd8-4180-9542-1020e83a704d\") " Sep 10 16:11:19.380929 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:19.380931 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j8c2v\" (UniqueName: \"kubernetes.io/projected/ecfb8c74-1806-43d3-8d8d-573617251bb9-kube-api-access-j8c2v\") pod \"ecfb8c74-1806-43d3-8d8d-573617251bb9\" (UID: \"ecfb8c74-1806-43d3-8d8d-573617251bb9\") " Sep 10 16:11:19.381167 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:19.380956 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6c987d0-fdd8-4180-9542-1020e83a704d-plugins-config-volume\") pod \"c6c987d0-fdd8-4180-9542-1020e83a704d\" (UID: \"c6c987d0-fdd8-4180-9542-1020e83a704d\") " Sep 10 16:11:19.381167 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:19.381009 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecfb8c74-1806-43d3-8d8d-573617251bb9-plugins-config-volume\") pod \"ecfb8c74-1806-43d3-8d8d-573617251bb9\" (UID: \"ecfb8c74-1806-43d3-8d8d-573617251bb9\") " Sep 10 16:11:19.381441 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:19.381405 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6c987d0-fdd8-4180-9542-1020e83a704d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c6c987d0-fdd8-4180-9542-1020e83a704d" (UID: "c6c987d0-fdd8-4180-9542-1020e83a704d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:11:19.381562 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:19.381445 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecfb8c74-1806-43d3-8d8d-573617251bb9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ecfb8c74-1806-43d3-8d8d-573617251bb9" (UID: "ecfb8c74-1806-43d3-8d8d-573617251bb9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:11:19.383344 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:19.383309 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecfb8c74-1806-43d3-8d8d-573617251bb9-kube-api-access-j8c2v" (OuterVolumeSpecName: "kube-api-access-j8c2v") pod "ecfb8c74-1806-43d3-8d8d-573617251bb9" (UID: "ecfb8c74-1806-43d3-8d8d-573617251bb9"). InnerVolumeSpecName "kube-api-access-j8c2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:11:19.383486 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:19.383455 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6c987d0-fdd8-4180-9542-1020e83a704d-kube-api-access-fz4cw" (OuterVolumeSpecName: "kube-api-access-fz4cw") pod "c6c987d0-fdd8-4180-9542-1020e83a704d" (UID: "c6c987d0-fdd8-4180-9542-1020e83a704d"). InnerVolumeSpecName "kube-api-access-fz4cw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:11:19.469186 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:19.469086 2576 generic.go:356] "Generic (PLEG): container finished" podID="c6c987d0-fdd8-4180-9542-1020e83a704d" containerID="de2dfe232787b7b0788756e1a9e5aa9b0fd67ed6509bfee7057fb71a48ec250b" exitCode=0 Sep 10 16:11:19.469186 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:19.469140 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-50ed73-657b6f9cdb-4xh7x" Sep 10 16:11:19.469186 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:19.469166 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-50ed73-657b6f9cdb-4xh7x" event={"ID":"c6c987d0-fdd8-4180-9542-1020e83a704d","Type":"ContainerDied","Data":"de2dfe232787b7b0788756e1a9e5aa9b0fd67ed6509bfee7057fb71a48ec250b"} Sep 10 16:11:19.469468 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:19.469220 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-50ed73-657b6f9cdb-4xh7x" event={"ID":"c6c987d0-fdd8-4180-9542-1020e83a704d","Type":"ContainerDied","Data":"126b1e502aa42dd24159620d83eb552aa2202c22e12bdc19d231b58fa727b47c"} Sep 10 16:11:19.469468 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:19.469250 2576 scope.go:117] "RemoveContainer" containerID="de2dfe232787b7b0788756e1a9e5aa9b0fd67ed6509bfee7057fb71a48ec250b" Sep 10 16:11:19.470820 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:19.470795 2576 generic.go:356] "Generic (PLEG): container finished" podID="ecfb8c74-1806-43d3-8d8d-573617251bb9" containerID="bebe906e4d8dcc38a67a9310429fce2f20de4d17b5e98f9434fbdad6a7a9f573" exitCode=0 Sep 10 16:11:19.471125 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:19.470840 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-50ed73-7c8bb55487-kpbvm" Sep 10 16:11:19.471125 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:19.470851 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-50ed73-7c8bb55487-kpbvm" event={"ID":"ecfb8c74-1806-43d3-8d8d-573617251bb9","Type":"ContainerDied","Data":"bebe906e4d8dcc38a67a9310429fce2f20de4d17b5e98f9434fbdad6a7a9f573"} Sep 10 16:11:19.471125 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:19.470903 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-50ed73-7c8bb55487-kpbvm" event={"ID":"ecfb8c74-1806-43d3-8d8d-573617251bb9","Type":"ContainerDied","Data":"67a77ef6fb0cf38c0dfb71b3acd89f81da2656f12c87808dcfe4a783fa44f621"} Sep 10 16:11:19.479165 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:19.479146 2576 scope.go:117] "RemoveContainer" containerID="de2dfe232787b7b0788756e1a9e5aa9b0fd67ed6509bfee7057fb71a48ec250b" Sep 10 16:11:19.479503 ip-10-0-139-236 kubenswrapper[2576]: E0910 16:11:19.479471 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de2dfe232787b7b0788756e1a9e5aa9b0fd67ed6509bfee7057fb71a48ec250b\": container with ID starting with de2dfe232787b7b0788756e1a9e5aa9b0fd67ed6509bfee7057fb71a48ec250b not found: ID does not exist" containerID="de2dfe232787b7b0788756e1a9e5aa9b0fd67ed6509bfee7057fb71a48ec250b" Sep 10 16:11:19.479573 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:19.479507 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de2dfe232787b7b0788756e1a9e5aa9b0fd67ed6509bfee7057fb71a48ec250b"} err="failed to get container status \"de2dfe232787b7b0788756e1a9e5aa9b0fd67ed6509bfee7057fb71a48ec250b\": rpc error: code = NotFound desc = could not find container \"de2dfe232787b7b0788756e1a9e5aa9b0fd67ed6509bfee7057fb71a48ec250b\": container with ID starting with de2dfe232787b7b0788756e1a9e5aa9b0fd67ed6509bfee7057fb71a48ec250b not found: ID does not exist" Sep 10 16:11:19.479573 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:19.479535 2576 scope.go:117] "RemoveContainer" containerID="bebe906e4d8dcc38a67a9310429fce2f20de4d17b5e98f9434fbdad6a7a9f573" Sep 10 16:11:19.481955 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:19.481934 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz4cw\" (UniqueName: \"kubernetes.io/projected/c6c987d0-fdd8-4180-9542-1020e83a704d-kube-api-access-fz4cw\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:11:19.481955 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:19.481954 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8c2v\" (UniqueName: \"kubernetes.io/projected/ecfb8c74-1806-43d3-8d8d-573617251bb9-kube-api-access-j8c2v\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:11:19.482118 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:19.481964 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6c987d0-fdd8-4180-9542-1020e83a704d-plugins-config-volume\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:11:19.482118 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:19.481972 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecfb8c74-1806-43d3-8d8d-573617251bb9-plugins-config-volume\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:11:19.488134 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:19.488117 2576 scope.go:117] "RemoveContainer" containerID="bebe906e4d8dcc38a67a9310429fce2f20de4d17b5e98f9434fbdad6a7a9f573" Sep 10 16:11:19.488439 ip-10-0-139-236 kubenswrapper[2576]: E0910 16:11:19.488419 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bebe906e4d8dcc38a67a9310429fce2f20de4d17b5e98f9434fbdad6a7a9f573\": container with ID starting with bebe906e4d8dcc38a67a9310429fce2f20de4d17b5e98f9434fbdad6a7a9f573 not found: ID does not exist" containerID="bebe906e4d8dcc38a67a9310429fce2f20de4d17b5e98f9434fbdad6a7a9f573" Sep 10 16:11:19.488482 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:19.488448 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bebe906e4d8dcc38a67a9310429fce2f20de4d17b5e98f9434fbdad6a7a9f573"} err="failed to get container status \"bebe906e4d8dcc38a67a9310429fce2f20de4d17b5e98f9434fbdad6a7a9f573\": rpc error: code = NotFound desc = could not find container \"bebe906e4d8dcc38a67a9310429fce2f20de4d17b5e98f9434fbdad6a7a9f573\": container with ID starting with bebe906e4d8dcc38a67a9310429fce2f20de4d17b5e98f9434fbdad6a7a9f573 not found: ID does not exist" Sep 10 16:11:19.497533 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:19.497505 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-50ed73-7c8bb55487-kpbvm"] Sep 10 16:11:19.519379 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:19.519341 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-50ed73-7c8bb55487-kpbvm"] Sep 10 16:11:19.540497 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:19.540462 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-50ed73-657b6f9cdb-4xh7x"] Sep 10 16:11:19.554939 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:19.554904 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-50ed73-657b6f9cdb-4xh7x"] Sep 10 16:11:19.715430 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:19.715387 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:11:21.169301 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:21.169265 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6c987d0-fdd8-4180-9542-1020e83a704d" path="/var/lib/kubelet/pods/c6c987d0-fdd8-4180-9542-1020e83a704d/volumes" Sep 10 16:11:21.169690 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:21.169631 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecfb8c74-1806-43d3-8d8d-573617251bb9" path="/var/lib/kubelet/pods/ecfb8c74-1806-43d3-8d8d-573617251bb9/volumes" Sep 10 16:11:21.981380 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:21.981342 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 16:11:26.003085 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:26.003050 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:11:26.082309 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:26.082267 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:11:26.298017 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:26.297979 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:11:26.599206 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:26.599117 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-67df2b-zr4bw"] Sep 10 16:11:26.599528 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:26.599516 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19052e67-3c56-4052-b5be-05e93c3cdd69" containerName="istio-proxy" Sep 10 16:11:26.599573 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:26.599530 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="19052e67-3c56-4052-b5be-05e93c3cdd69" containerName="istio-proxy" Sep 10 16:11:26.599573 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:26.599546 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6c987d0-fdd8-4180-9542-1020e83a704d" containerName="payload-pre-processing" Sep 10 16:11:26.599573 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:26.599552 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6c987d0-fdd8-4180-9542-1020e83a704d" containerName="payload-pre-processing" Sep 10 16:11:26.599573 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:26.599559 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecfb8c74-1806-43d3-8d8d-573617251bb9" containerName="payload-processing" Sep 10 16:11:26.599573 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:26.599564 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecfb8c74-1806-43d3-8d8d-573617251bb9" containerName="payload-processing" Sep 10 16:11:26.599573 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:26.599570 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d80d4de-29ec-4b05-899d-7f9c9e6778be" containerName="payload-pre-processing" Sep 10 16:11:26.599803 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:26.599575 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d80d4de-29ec-4b05-899d-7f9c9e6778be" containerName="payload-pre-processing" Sep 10 16:11:26.599803 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:26.599581 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eacc589c-de4a-4176-8847-616bc6af0834" containerName="revoke-keys" Sep 10 16:11:26.599803 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:26.599586 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="eacc589c-de4a-4176-8847-616bc6af0834" containerName="revoke-keys" Sep 10 16:11:26.599803 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:26.599594 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19052e67-3c56-4052-b5be-05e93c3cdd69" containerName="istio-proxy" Sep 10 16:11:26.599803 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:26.599600 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="19052e67-3c56-4052-b5be-05e93c3cdd69" containerName="istio-proxy" Sep 10 16:11:26.599803 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:26.599658 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="19052e67-3c56-4052-b5be-05e93c3cdd69" containerName="istio-proxy" Sep 10 16:11:26.599803 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:26.599665 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d80d4de-29ec-4b05-899d-7f9c9e6778be" containerName="payload-pre-processing" Sep 10 16:11:26.599803 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:26.599672 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="eacc589c-de4a-4176-8847-616bc6af0834" containerName="revoke-keys" Sep 10 16:11:26.599803 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:26.599678 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6c987d0-fdd8-4180-9542-1020e83a704d" containerName="payload-pre-processing" Sep 10 16:11:26.599803 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:26.599686 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecfb8c74-1806-43d3-8d8d-573617251bb9" containerName="payload-processing" Sep 10 16:11:26.604231 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:26.604208 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-67df2b-zr4bw" Sep 10 16:11:26.606795 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:26.606772 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwwht\"" Sep 10 16:11:26.606924 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:26.606778 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 16:11:26.610985 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:26.610956 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-67df2b-zr4bw"] Sep 10 16:11:26.652672 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:26.652617 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbed4662-27b5-4c79-821e-c6217e0dd14a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-67df2b-zr4bw\" (UID: \"dbed4662-27b5-4c79-821e-c6217e0dd14a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-67df2b-zr4bw" Sep 10 16:11:26.754175 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:26.754131 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbed4662-27b5-4c79-821e-c6217e0dd14a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-67df2b-zr4bw\" (UID: \"dbed4662-27b5-4c79-821e-c6217e0dd14a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-67df2b-zr4bw" Sep 10 16:11:26.754842 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:26.754820 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbed4662-27b5-4c79-821e-c6217e0dd14a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-67df2b-zr4bw\" (UID: \"dbed4662-27b5-4c79-821e-c6217e0dd14a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-67df2b-zr4bw" Sep 10 16:11:26.836683 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:26.836648 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:11:26.917644 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:26.917531 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-67df2b-zr4bw" Sep 10 16:11:27.193543 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:27.193516 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-67df2b-zr4bw"] Sep 10 16:11:27.195895 ip-10-0-139-236 kubenswrapper[2576]: W0910 16:11:27.195864 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbed4662_27b5_4c79_821e_c6217e0dd14a.slice/crio-614a040dad79002cdb73c7cae8f8f03218d945073f23ab0974ad02116ddf43d1 WatchSource:0}: Error finding container 614a040dad79002cdb73c7cae8f8f03218d945073f23ab0974ad02116ddf43d1: Status 404 returned error can't find the container with id 614a040dad79002cdb73c7cae8f8f03218d945073f23ab0974ad02116ddf43d1 Sep 10 16:11:27.519145 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:27.519104 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-67df2b-zr4bw" event={"ID":"dbed4662-27b5-4c79-821e-c6217e0dd14a","Type":"ContainerStarted","Data":"30992a809fe065a0309332095b5d51b389e4e0d3fbfc644e35f8d8a8b6024837"} Sep 10 16:11:27.519344 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:27.519171 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-67df2b-zr4bw" event={"ID":"dbed4662-27b5-4c79-821e-c6217e0dd14a","Type":"ContainerStarted","Data":"614a040dad79002cdb73c7cae8f8f03218d945073f23ab0974ad02116ddf43d1"} Sep 10 16:11:27.536082 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:27.536023 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-67df2b-zr4bw" podStartSLOduration=1.5360052830000002 podStartE2EDuration="1.536005283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:11:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:11:27.533981881 +0000 UTC m=+1764.887501044" watchObservedRunningTime="2026-09-10 16:11:27.536005283 +0000 UTC m=+1764.889524445" Sep 10 16:11:28.526553 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:28.526521 2576 generic.go:356] "Generic (PLEG): container finished" podID="dbed4662-27b5-4c79-821e-c6217e0dd14a" containerID="30992a809fe065a0309332095b5d51b389e4e0d3fbfc644e35f8d8a8b6024837" exitCode=0 Sep 10 16:11:28.527009 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:28.526603 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-67df2b-zr4bw" event={"ID":"dbed4662-27b5-4c79-821e-c6217e0dd14a","Type":"ContainerDied","Data":"30992a809fe065a0309332095b5d51b389e4e0d3fbfc644e35f8d8a8b6024837"} Sep 10 16:11:29.674835 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:29.674799 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-67df2b-zr4bw" Sep 10 16:11:29.785074 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:29.784959 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbed4662-27b5-4c79-821e-c6217e0dd14a-maas-api-service-ca\") pod \"dbed4662-27b5-4c79-821e-c6217e0dd14a\" (UID: \"dbed4662-27b5-4c79-821e-c6217e0dd14a\") " Sep 10 16:11:29.785343 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:29.785321 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbed4662-27b5-4c79-821e-c6217e0dd14a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dbed4662-27b5-4c79-821e-c6217e0dd14a" (UID: "dbed4662-27b5-4c79-821e-c6217e0dd14a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:11:29.886289 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:29.886242 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbed4662-27b5-4c79-821e-c6217e0dd14a-maas-api-service-ca\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:11:30.539239 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:30.539201 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-67df2b-zr4bw" Sep 10 16:11:30.539239 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:30.539225 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-67df2b-zr4bw" event={"ID":"dbed4662-27b5-4c79-821e-c6217e0dd14a","Type":"ContainerDied","Data":"614a040dad79002cdb73c7cae8f8f03218d945073f23ab0974ad02116ddf43d1"} Sep 10 16:11:30.539465 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:30.539276 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="614a040dad79002cdb73c7cae8f8f03218d945073f23ab0974ad02116ddf43d1" Sep 10 16:11:37.274705 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:37.274644 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:11:37.320656 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:37.320624 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-67df2b-6cbf8787ff-rdh76"] Sep 10 16:11:37.320884 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:37.320862 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-67df2b-6cbf8787ff-rdh76" podUID="1dec1a26-3316-4023-ba38-d339a73e4549" containerName="payload-processing" containerID="cri-o://a90dc05c01038c6ab9386ab4454c2ecb4fe64dca8835c3509dd5f26f215f8b6a" gracePeriod=30 Sep 10 16:11:37.371910 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:37.371863 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:11:37.477007 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:37.476971 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:11:37.526992 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:37.526904 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-67df2b-zr4bw"] Sep 10 16:11:37.535704 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:37.535670 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-67df2b-zr4bw"] Sep 10 16:11:37.577858 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:37.577826 2576 generic.go:356] "Generic (PLEG): container finished" podID="1dec1a26-3316-4023-ba38-d339a73e4549" containerID="a90dc05c01038c6ab9386ab4454c2ecb4fe64dca8835c3509dd5f26f215f8b6a" exitCode=0 Sep 10 16:11:37.578018 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:37.577906 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-67df2b-6cbf8787ff-rdh76" event={"ID":"1dec1a26-3316-4023-ba38-d339a73e4549","Type":"ContainerDied","Data":"a90dc05c01038c6ab9386ab4454c2ecb4fe64dca8835c3509dd5f26f215f8b6a"} Sep 10 16:11:37.584470 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:37.584415 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:11:37.595438 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:37.595402 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-67df2b-6cbf8787ff-rdh76" Sep 10 16:11:37.761157 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:37.761113 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j6wqk\" (UniqueName: \"kubernetes.io/projected/1dec1a26-3316-4023-ba38-d339a73e4549-kube-api-access-j6wqk\") pod \"1dec1a26-3316-4023-ba38-d339a73e4549\" (UID: \"1dec1a26-3316-4023-ba38-d339a73e4549\") " Sep 10 16:11:37.761157 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:37.761168 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dec1a26-3316-4023-ba38-d339a73e4549-plugins-config-volume\") pod \"1dec1a26-3316-4023-ba38-d339a73e4549\" (UID: \"1dec1a26-3316-4023-ba38-d339a73e4549\") " Sep 10 16:11:37.761561 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:37.761534 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dec1a26-3316-4023-ba38-d339a73e4549-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1dec1a26-3316-4023-ba38-d339a73e4549" (UID: "1dec1a26-3316-4023-ba38-d339a73e4549"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:11:37.763502 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:37.763482 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dec1a26-3316-4023-ba38-d339a73e4549-kube-api-access-j6wqk" (OuterVolumeSpecName: "kube-api-access-j6wqk") pod "1dec1a26-3316-4023-ba38-d339a73e4549" (UID: "1dec1a26-3316-4023-ba38-d339a73e4549"). InnerVolumeSpecName "kube-api-access-j6wqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:11:37.862765 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:37.862682 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6wqk\" (UniqueName: \"kubernetes.io/projected/1dec1a26-3316-4023-ba38-d339a73e4549-kube-api-access-j6wqk\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:11:37.862765 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:37.862756 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dec1a26-3316-4023-ba38-d339a73e4549-plugins-config-volume\") on node \"ip-10-0-139-236.ec2.internal\" DevicePath \"\"" Sep 10 16:11:38.046093 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:38.046054 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:11:38.585171 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:38.585140 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-67df2b-6cbf8787ff-rdh76" Sep 10 16:11:38.585686 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:38.585142 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-67df2b-6cbf8787ff-rdh76" event={"ID":"1dec1a26-3316-4023-ba38-d339a73e4549","Type":"ContainerDied","Data":"c75923137ab24fcda23899e563d34eaa7c9491cc792046e06677ed02a7955d36"} Sep 10 16:11:38.585686 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:38.585310 2576 scope.go:117] "RemoveContainer" containerID="a90dc05c01038c6ab9386ab4454c2ecb4fe64dca8835c3509dd5f26f215f8b6a" Sep 10 16:11:38.606376 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:38.606345 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-67df2b-6cbf8787ff-rdh76"] Sep 10 16:11:38.613586 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:38.613553 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-67df2b-6cbf8787ff-rdh76"] Sep 10 16:11:39.169171 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:39.169137 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1dec1a26-3316-4023-ba38-d339a73e4549" path="/var/lib/kubelet/pods/1dec1a26-3316-4023-ba38-d339a73e4549/volumes" Sep 10 16:11:39.169488 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:39.169476 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbed4662-27b5-4c79-821e-c6217e0dd14a" path="/var/lib/kubelet/pods/dbed4662-27b5-4c79-821e-c6217e0dd14a/volumes" Sep 10 16:11:40.088308 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:40.088262 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 16:11:44.614863 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:44.614821 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:11:44.705895 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:44.705852 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:11:44.933659 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:44.933564 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:11:49.921281 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:11:49.921241 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 16:13:13.670928 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:13:13.670887 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:13:14.824321 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:13:14.824282 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 16:13:15.243070 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:13:15.242972 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:13:16.663211 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:13:16.663177 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:13:40.890436 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:13:40.890346 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 16:13:48.788539 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:13:48.788496 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 16:13:56.582454 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:13:56.582422 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 16:14:02.978470 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:14:02.978432 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 16:14:15.089886 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:14:15.089851 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 16:14:25.080327 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:14:25.080292 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 16:14:30.045769 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:14:30.045628 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:14:30.100362 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:14:30.100323 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:15:00.140358 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:15:00.140272 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817615-tvx6b"] Sep 10 16:15:00.140822 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:15:00.140787 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dec1a26-3316-4023-ba38-d339a73e4549" containerName="payload-processing" Sep 10 16:15:00.140822 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:15:00.140808 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dec1a26-3316-4023-ba38-d339a73e4549" containerName="payload-processing" Sep 10 16:15:00.140923 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:15:00.140828 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbed4662-27b5-4c79-821e-c6217e0dd14a" containerName="revoke-keys" Sep 10 16:15:00.140923 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:15:00.140837 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbed4662-27b5-4c79-821e-c6217e0dd14a" containerName="revoke-keys" Sep 10 16:15:00.140987 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:15:00.140928 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbed4662-27b5-4c79-821e-c6217e0dd14a" containerName="revoke-keys" Sep 10 16:15:00.140987 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:15:00.140940 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dec1a26-3316-4023-ba38-d339a73e4549" containerName="payload-processing" Sep 10 16:15:00.140987 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:15:00.140947 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="19052e67-3c56-4052-b5be-05e93c3cdd69" containerName="istio-proxy" Sep 10 16:15:00.144016 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:15:00.143998 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817615-tvx6b" Sep 10 16:15:00.146165 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:15:00.146143 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwwht\"" Sep 10 16:15:00.150383 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:15:00.150362 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817615-tvx6b"] Sep 10 16:15:00.155604 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:15:00.155587 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817615-tvx6b" Sep 10 16:15:00.430348 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:15:00.430321 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817615-tvx6b"] Sep 10 16:15:00.432823 ip-10-0-139-236 kubenswrapper[2576]: W0910 16:15:00.432790 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa77af9c_3eda_4a67_9128_ba054ceb6f4f.slice/crio-0884e7af439738e3f1dbe6c1d4f39bb90623dca8e1d8969571dff339ea077566 WatchSource:0}: Error finding container 0884e7af439738e3f1dbe6c1d4f39bb90623dca8e1d8969571dff339ea077566: Status 404 returned error can't find the container with id 0884e7af439738e3f1dbe6c1d4f39bb90623dca8e1d8969571dff339ea077566 Sep 10 16:15:00.598456 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:15:00.598423 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817615-tvx6b" event={"ID":"aa77af9c-3eda-4a67-9128-ba054ceb6f4f","Type":"ContainerStarted","Data":"0884e7af439738e3f1dbe6c1d4f39bb90623dca8e1d8969571dff339ea077566"} Sep 10 16:15:01.605557 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:15:01.605514 2576 generic.go:356] "Generic (PLEG): container finished" podID="aa77af9c-3eda-4a67-9128-ba054ceb6f4f" containerID="52ac93ef0dd4395b172f9de072a814d221680d089d1a68afb41efdae159fc732" exitCode=0 Sep 10 16:15:01.605975 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:15:01.605578 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817615-tvx6b" event={"ID":"aa77af9c-3eda-4a67-9128-ba054ceb6f4f","Type":"ContainerDied","Data":"52ac93ef0dd4395b172f9de072a814d221680d089d1a68afb41efdae159fc732"} Sep 10 16:15:02.741084 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:15:02.741061 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817615-tvx6b" Sep 10 16:15:03.615708 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:15:03.615684 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817615-tvx6b" Sep 10 16:15:03.615881 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:15:03.615710 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817615-tvx6b" event={"ID":"aa77af9c-3eda-4a67-9128-ba054ceb6f4f","Type":"ContainerDied","Data":"0884e7af439738e3f1dbe6c1d4f39bb90623dca8e1d8969571dff339ea077566"} Sep 10 16:15:03.615881 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:15:03.615774 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0884e7af439738e3f1dbe6c1d4f39bb90623dca8e1d8969571dff339ea077566" Sep 10 16:15:05.744532 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:15:05.744500 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:15:07.417689 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:15:07.417629 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:15:52.098103 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:15:52.098069 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 16:16:00.270219 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:16:00.270186 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 16:16:03.584832 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:16:03.584800 2576 scope.go:117] "RemoveContainer" containerID="aff8df95666b130cdc2cefbe0157f3ffa020ac5907e17619af89b761c281b556" Sep 10 16:16:06.296114 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:16:06.296077 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:16:08.413548 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:16:08.413513 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 16:16:10.749093 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:16:10.749062 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:16:19.094204 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:16:19.094171 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 16:16:31.643999 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:16:31.643896 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:16:40.083879 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:16:40.083827 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:16:41.548117 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:16:41.548079 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:16:43.379040 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:16:43.379001 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 16:16:51.885251 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:16:51.885213 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 16:16:59.781277 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:16:59.781241 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 16:17:03.603436 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:03.603402 2576 scope.go:117] "RemoveContainer" containerID="1118831b6d3af2fb3e42cea039dff78260eadf60b43bed102ff47aa33ba987e3" Sep 10 16:17:07.107188 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:07.107152 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 16:17:14.018699 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:14.018666 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 16:17:19.931385 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:19.931345 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 16:17:27.188467 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:27.188438 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-89xpd"] Sep 10 16:17:34.927630 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:34.927599 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-n9wld_da14561b-2f8c-4987-953f-260f78e954db/manager/0.log" Sep 10 16:17:35.048532 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:35.048504 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-w7jnk_bfd89460-9cbb-4808-ad22-8b14f6e690ea/manager/0.log" Sep 10 16:17:35.296757 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:35.296709 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-pn66h_75f36c0a-f51b-4ee6-8917-f9153bfa9f2d/server/0.log" Sep 10 16:17:35.422247 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:35.422220 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-8h7gh_fc2d768d-cf98-4412-a63c-1ad1c466cdc5/manager/0.log" Sep 10 16:17:35.542329 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:35.542307 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-97f7948ff-cr5vg_899506ad-9415-4a67-b953-3d8134d64297/copy-manifests/0.log" Sep 10 16:17:35.554841 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:35.554779 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-97f7948ff-cr5vg_899506ad-9415-4a67-b953-3d8134d64297/manager/0.log" Sep 10 16:17:36.511722 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:36.511694 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29817615-tvx6b_aa77af9c-3eda-4a67-9128-ba054ceb6f4f/cleanup/0.log" Sep 10 16:17:38.016921 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:38.016893 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-89xpd_68c81a2d-967e-4f60-a6f7-8ecec49dcecc/limitador/0.log" Sep 10 16:17:38.143207 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:38.143179 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-s25wn_3a136beb-6a6c-420b-9d5a-87835dc85964/manager/0.log" Sep 10 16:17:38.493192 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:38.493164 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz_df543b0d-1120-499b-8a4a-ea71ca5e539d/istio-proxy/0.log" Sep 10 16:17:38.614181 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:38.614157 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-4xws7_eff52c89-238c-4887-b268-2bf69a3fabab/discovery/0.log" Sep 10 16:17:38.837697 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:38.837675 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-644bb569f8-dslcn_0b5650cc-b071-4c0a-a616-16012fb0a0a0/kube-auth-proxy/0.log" Sep 10 16:17:38.959409 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:38.959384 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-fq9cv_0b754538-f8b9-4891-aa4f-b910ddd9d35e/istio-proxy/1.log" Sep 10 16:17:39.081483 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:39.081456 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-9g9rc_b5c2d260-e0f6-4e9d-a217-1c4ada9c1ade/payload-pre-processing/0.log" Sep 10 16:17:40.241702 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:40.241667 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-r6klk_cc071832-44c6-42a3-ba2d-d9a9f98afbc8/main/0.log" Sep 10 16:17:45.826966 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:45.826935 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-t86vd/must-gather-rq6xc"] Sep 10 16:17:45.827324 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:45.827311 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa77af9c-3eda-4a67-9128-ba054ceb6f4f" containerName="cleanup" Sep 10 16:17:45.827324 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:45.827323 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa77af9c-3eda-4a67-9128-ba054ceb6f4f" containerName="cleanup" Sep 10 16:17:45.827398 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:45.827391 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa77af9c-3eda-4a67-9128-ba054ceb6f4f" containerName="cleanup" Sep 10 16:17:45.830496 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:45.830473 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t86vd/must-gather-rq6xc" Sep 10 16:17:45.835502 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:45.835480 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t86vd\"/\"kube-root-ca.crt\"" Sep 10 16:17:45.836102 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:45.836077 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-t86vd\"/\"default-dockercfg-b47qk\"" Sep 10 16:17:45.836227 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:45.836077 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t86vd\"/\"openshift-service-ca.crt\"" Sep 10 16:17:45.852075 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:45.852053 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t86vd/must-gather-rq6xc"] Sep 10 16:17:45.856426 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:45.856402 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:17:45.902191 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:45.902166 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmsdm\" (UniqueName: \"kubernetes.io/projected/bb57de0c-9360-4f36-95cb-f84cf705ac0d-kube-api-access-pmsdm\") pod \"must-gather-rq6xc\" (UID: \"bb57de0c-9360-4f36-95cb-f84cf705ac0d\") " pod="openshift-must-gather-t86vd/must-gather-rq6xc" Sep 10 16:17:45.902280 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:45.902196 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bb57de0c-9360-4f36-95cb-f84cf705ac0d-must-gather-output\") pod \"must-gather-rq6xc\" (UID: \"bb57de0c-9360-4f36-95cb-f84cf705ac0d\") " pod="openshift-must-gather-t86vd/must-gather-rq6xc" Sep 10 16:17:45.963192 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:45.963163 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" limit=1024 Sep 10 16:17:46.003488 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:46.003468 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmsdm\" (UniqueName: \"kubernetes.io/projected/bb57de0c-9360-4f36-95cb-f84cf705ac0d-kube-api-access-pmsdm\") pod \"must-gather-rq6xc\" (UID: \"bb57de0c-9360-4f36-95cb-f84cf705ac0d\") " pod="openshift-must-gather-t86vd/must-gather-rq6xc" Sep 10 16:17:46.003574 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:46.003497 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bb57de0c-9360-4f36-95cb-f84cf705ac0d-must-gather-output\") pod \"must-gather-rq6xc\" (UID: \"bb57de0c-9360-4f36-95cb-f84cf705ac0d\") " pod="openshift-must-gather-t86vd/must-gather-rq6xc" Sep 10 16:17:46.003819 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:46.003804 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bb57de0c-9360-4f36-95cb-f84cf705ac0d-must-gather-output\") pod \"must-gather-rq6xc\" (UID: \"bb57de0c-9360-4f36-95cb-f84cf705ac0d\") " pod="openshift-must-gather-t86vd/must-gather-rq6xc" Sep 10 16:17:46.012952 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:46.012934 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmsdm\" (UniqueName: \"kubernetes.io/projected/bb57de0c-9360-4f36-95cb-f84cf705ac0d-kube-api-access-pmsdm\") pod \"must-gather-rq6xc\" (UID: \"bb57de0c-9360-4f36-95cb-f84cf705ac0d\") " pod="openshift-must-gather-t86vd/must-gather-rq6xc" Sep 10 16:17:46.139845 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:46.139795 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t86vd/must-gather-rq6xc" Sep 10 16:17:46.435240 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:46.435219 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t86vd/must-gather-rq6xc"] Sep 10 16:17:46.436855 ip-10-0-139-236 kubenswrapper[2576]: W0910 16:17:46.436828 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb57de0c_9360_4f36_95cb_f84cf705ac0d.slice/crio-6d5e03ac61ad20f87e2bab52f0a657fe045d60140f0a91b60affbda78d37d81a WatchSource:0}: Error finding container 6d5e03ac61ad20f87e2bab52f0a657fe045d60140f0a91b60affbda78d37d81a: Status 404 returned error can't find the container with id 6d5e03ac61ad20f87e2bab52f0a657fe045d60140f0a91b60affbda78d37d81a Sep 10 16:17:46.439063 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:46.439040 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 16:17:47.397933 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:47.397881 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t86vd/must-gather-rq6xc" event={"ID":"bb57de0c-9360-4f36-95cb-f84cf705ac0d","Type":"ContainerStarted","Data":"1c811e76e1c7d55012e391e0bb450f1595b202ad5ae15f5901800e15652e08da"} Sep 10 16:17:47.398306 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:47.397953 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t86vd/must-gather-rq6xc" event={"ID":"bb57de0c-9360-4f36-95cb-f84cf705ac0d","Type":"ContainerStarted","Data":"6d5e03ac61ad20f87e2bab52f0a657fe045d60140f0a91b60affbda78d37d81a"} Sep 10 16:17:48.406758 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:48.406709 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t86vd/must-gather-rq6xc" event={"ID":"bb57de0c-9360-4f36-95cb-f84cf705ac0d","Type":"ContainerStarted","Data":"7a8345c4b0565504358f7f9a44e0592979114c8ba7bdf6304bc2cb1d09526d7a"} Sep 10 16:17:48.429338 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:48.429290 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-t86vd/must-gather-rq6xc" podStartSLOduration=2.58074544 podStartE2EDuration="3.429274294s" totalImagesPullingTime="848.528854ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:17:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:17:48.427964751 +0000 UTC m=+2145.781483916" watchObservedRunningTime="2026-09-10 16:17:48.429274294 +0000 UTC m=+2145.782793455" Sep 10 16:17:48.926756 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:48.926702 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-hzjqc_1b1f777f-8ed4-408b-9721-80c9ae47af6e/global-pull-secret-syncer/0.log" Sep 10 16:17:49.090788 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:49.090754 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-5sgqq_a2e3e295-41d8-4b2f-b0a4-26ec4ec7d11a/konnectivity-agent/0.log" Sep 10 16:17:49.249599 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:49.249513 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-236.ec2.internal_c089abe4aa32d6992a9466be84e92f1f/haproxy/0.log" Sep 10 16:17:53.751119 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:53.751090 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-89xpd_68c81a2d-967e-4f60-a6f7-8ecec49dcecc/limitador/0.log" Sep 10 16:17:53.835603 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:53.835568 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-s25wn_3a136beb-6a6c-420b-9d5a-87835dc85964/manager/0.log" Sep 10 16:17:55.416011 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:55.415979 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-bw9q7_0a661843-e27a-4388-b301-49c9eafddf18/kube-state-metrics/0.log" Sep 10 16:17:55.436103 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:55.436032 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-bw9q7_0a661843-e27a-4388-b301-49c9eafddf18/kube-rbac-proxy-main/0.log" Sep 10 16:17:55.462951 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:55.462915 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-bw9q7_0a661843-e27a-4388-b301-49c9eafddf18/kube-rbac-proxy-self/0.log" Sep 10 16:17:55.491606 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:55.491524 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6c668cb757-6pn84_494d55d5-ac80-4d7c-b9d1-a9541d59924f/metrics-server/0.log" Sep 10 16:17:55.771417 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:55.771333 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-k8jsb_d78d8162-f3d9-4730-8354-e8eb6d8e676c/node-exporter/0.log" Sep 10 16:17:55.797017 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:55.796984 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-k8jsb_d78d8162-f3d9-4730-8354-e8eb6d8e676c/kube-rbac-proxy/0.log" Sep 10 16:17:55.821958 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:55.821924 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-k8jsb_d78d8162-f3d9-4730-8354-e8eb6d8e676c/init-textfile/0.log" Sep 10 16:17:55.868781 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:55.868700 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-tz8fc_566fd587-ad31-49c7-8480-62ace5e8c065/kube-rbac-proxy-main/0.log" Sep 10 16:17:55.895802 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:55.895699 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-tz8fc_566fd587-ad31-49c7-8480-62ace5e8c065/kube-rbac-proxy-self/0.log" Sep 10 16:17:55.920940 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:55.920908 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-tz8fc_566fd587-ad31-49c7-8480-62ace5e8c065/openshift-state-metrics/0.log" Sep 10 16:17:55.967823 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:55.967794 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_91f948cc-e301-4b86-a1e8-a74ac8fba5ed/prometheus/0.log" Sep 10 16:17:56.009209 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:56.009169 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_91f948cc-e301-4b86-a1e8-a74ac8fba5ed/config-reloader/0.log" Sep 10 16:17:56.037073 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:56.037045 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_91f948cc-e301-4b86-a1e8-a74ac8fba5ed/thanos-sidecar/0.log" Sep 10 16:17:56.064752 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:56.064672 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_91f948cc-e301-4b86-a1e8-a74ac8fba5ed/kube-rbac-proxy-web/0.log" Sep 10 16:17:56.094801 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:56.094761 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_91f948cc-e301-4b86-a1e8-a74ac8fba5ed/kube-rbac-proxy/0.log" Sep 10 16:17:56.127049 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:56.127018 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_91f948cc-e301-4b86-a1e8-a74ac8fba5ed/kube-rbac-proxy-thanos/0.log" Sep 10 16:17:56.150588 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:56.150555 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_91f948cc-e301-4b86-a1e8-a74ac8fba5ed/init-config-reloader/0.log" Sep 10 16:17:56.270016 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:56.269983 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5b66c47b87-khlqk_a9aa86db-f046-49a4-bba7-3a9ca2bdae16/telemeter-client/0.log" Sep 10 16:17:56.291719 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:56.291692 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5b66c47b87-khlqk_a9aa86db-f046-49a4-bba7-3a9ca2bdae16/reload/0.log" Sep 10 16:17:56.330918 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:56.330884 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5b66c47b87-khlqk_a9aa86db-f046-49a4-bba7-3a9ca2bdae16/kube-rbac-proxy/0.log" Sep 10 16:17:59.207022 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:59.206991 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-t86vd/perf-node-gather-daemonset-brwp4"] Sep 10 16:17:59.212022 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:59.212000 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t86vd/perf-node-gather-daemonset-brwp4" Sep 10 16:17:59.225144 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:59.225113 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t86vd/perf-node-gather-daemonset-brwp4"] Sep 10 16:17:59.353126 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:59.353092 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/28ba77da-40fa-4083-8b4e-531cc0a0e659-lib-modules\") pod \"perf-node-gather-daemonset-brwp4\" (UID: \"28ba77da-40fa-4083-8b4e-531cc0a0e659\") " pod="openshift-must-gather-t86vd/perf-node-gather-daemonset-brwp4" Sep 10 16:17:59.353126 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:59.353135 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/28ba77da-40fa-4083-8b4e-531cc0a0e659-proc\") pod \"perf-node-gather-daemonset-brwp4\" (UID: \"28ba77da-40fa-4083-8b4e-531cc0a0e659\") " pod="openshift-must-gather-t86vd/perf-node-gather-daemonset-brwp4" Sep 10 16:17:59.353374 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:59.353163 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/28ba77da-40fa-4083-8b4e-531cc0a0e659-podres\") pod \"perf-node-gather-daemonset-brwp4\" (UID: \"28ba77da-40fa-4083-8b4e-531cc0a0e659\") " pod="openshift-must-gather-t86vd/perf-node-gather-daemonset-brwp4" Sep 10 16:17:59.353374 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:59.353191 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m957\" (UniqueName: \"kubernetes.io/projected/28ba77da-40fa-4083-8b4e-531cc0a0e659-kube-api-access-2m957\") pod \"perf-node-gather-daemonset-brwp4\" (UID: \"28ba77da-40fa-4083-8b4e-531cc0a0e659\") " pod="openshift-must-gather-t86vd/perf-node-gather-daemonset-brwp4" Sep 10 16:17:59.353374 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:59.353302 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/28ba77da-40fa-4083-8b4e-531cc0a0e659-sys\") pod \"perf-node-gather-daemonset-brwp4\" (UID: \"28ba77da-40fa-4083-8b4e-531cc0a0e659\") " pod="openshift-must-gather-t86vd/perf-node-gather-daemonset-brwp4" Sep 10 16:17:59.454539 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:59.454503 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/28ba77da-40fa-4083-8b4e-531cc0a0e659-podres\") pod \"perf-node-gather-daemonset-brwp4\" (UID: \"28ba77da-40fa-4083-8b4e-531cc0a0e659\") " pod="openshift-must-gather-t86vd/perf-node-gather-daemonset-brwp4" Sep 10 16:17:59.454775 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:59.454552 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m957\" (UniqueName: \"kubernetes.io/projected/28ba77da-40fa-4083-8b4e-531cc0a0e659-kube-api-access-2m957\") pod \"perf-node-gather-daemonset-brwp4\" (UID: \"28ba77da-40fa-4083-8b4e-531cc0a0e659\") " pod="openshift-must-gather-t86vd/perf-node-gather-daemonset-brwp4" Sep 10 16:17:59.454775 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:59.454644 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/28ba77da-40fa-4083-8b4e-531cc0a0e659-sys\") pod \"perf-node-gather-daemonset-brwp4\" (UID: \"28ba77da-40fa-4083-8b4e-531cc0a0e659\") " pod="openshift-must-gather-t86vd/perf-node-gather-daemonset-brwp4" Sep 10 16:17:59.454775 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:59.454698 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/28ba77da-40fa-4083-8b4e-531cc0a0e659-podres\") pod \"perf-node-gather-daemonset-brwp4\" (UID: \"28ba77da-40fa-4083-8b4e-531cc0a0e659\") " pod="openshift-must-gather-t86vd/perf-node-gather-daemonset-brwp4" Sep 10 16:17:59.454775 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:59.454713 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/28ba77da-40fa-4083-8b4e-531cc0a0e659-lib-modules\") pod \"perf-node-gather-daemonset-brwp4\" (UID: \"28ba77da-40fa-4083-8b4e-531cc0a0e659\") " pod="openshift-must-gather-t86vd/perf-node-gather-daemonset-brwp4" Sep 10 16:17:59.455008 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:59.454804 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/28ba77da-40fa-4083-8b4e-531cc0a0e659-sys\") pod \"perf-node-gather-daemonset-brwp4\" (UID: \"28ba77da-40fa-4083-8b4e-531cc0a0e659\") " pod="openshift-must-gather-t86vd/perf-node-gather-daemonset-brwp4" Sep 10 16:17:59.455008 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:59.454860 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/28ba77da-40fa-4083-8b4e-531cc0a0e659-proc\") pod \"perf-node-gather-daemonset-brwp4\" (UID: \"28ba77da-40fa-4083-8b4e-531cc0a0e659\") " pod="openshift-must-gather-t86vd/perf-node-gather-daemonset-brwp4" Sep 10 16:17:59.455008 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:59.454908 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/28ba77da-40fa-4083-8b4e-531cc0a0e659-proc\") pod \"perf-node-gather-daemonset-brwp4\" (UID: \"28ba77da-40fa-4083-8b4e-531cc0a0e659\") " pod="openshift-must-gather-t86vd/perf-node-gather-daemonset-brwp4" Sep 10 16:17:59.455008 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:59.454865 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/28ba77da-40fa-4083-8b4e-531cc0a0e659-lib-modules\") pod \"perf-node-gather-daemonset-brwp4\" (UID: \"28ba77da-40fa-4083-8b4e-531cc0a0e659\") " pod="openshift-must-gather-t86vd/perf-node-gather-daemonset-brwp4" Sep 10 16:17:59.465412 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:59.465347 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m957\" (UniqueName: \"kubernetes.io/projected/28ba77da-40fa-4083-8b4e-531cc0a0e659-kube-api-access-2m957\") pod \"perf-node-gather-daemonset-brwp4\" (UID: \"28ba77da-40fa-4083-8b4e-531cc0a0e659\") " pod="openshift-must-gather-t86vd/perf-node-gather-daemonset-brwp4" Sep 10 16:17:59.524246 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:59.524014 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t86vd/perf-node-gather-daemonset-brwp4" Sep 10 16:17:59.835771 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:17:59.835698 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t86vd/perf-node-gather-daemonset-brwp4"] Sep 10 16:17:59.840307 ip-10-0-139-236 kubenswrapper[2576]: W0910 16:17:59.840262 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod28ba77da_40fa_4083_8b4e_531cc0a0e659.slice/crio-07e96344e91961bfe775a261977e5747cc2674a03ef411782bc296b149572b49 WatchSource:0}: Error finding container 07e96344e91961bfe775a261977e5747cc2674a03ef411782bc296b149572b49: Status 404 returned error can't find the container with id 07e96344e91961bfe775a261977e5747cc2674a03ef411782bc296b149572b49 Sep 10 16:18:00.489122 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:18:00.489072 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t86vd/perf-node-gather-daemonset-brwp4" event={"ID":"28ba77da-40fa-4083-8b4e-531cc0a0e659","Type":"ContainerStarted","Data":"a958b2800d751d04f3da5eb1f1414ca782502abd50fe010d8dfe7c197c4ea26d"} Sep 10 16:18:00.489583 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:18:00.489140 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t86vd/perf-node-gather-daemonset-brwp4" event={"ID":"28ba77da-40fa-4083-8b4e-531cc0a0e659","Type":"ContainerStarted","Data":"07e96344e91961bfe775a261977e5747cc2674a03ef411782bc296b149572b49"} Sep 10 16:18:00.489583 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:18:00.489322 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-t86vd/perf-node-gather-daemonset-brwp4" Sep 10 16:18:00.512947 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:18:00.512891 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-t86vd/perf-node-gather-daemonset-brwp4" podStartSLOduration=1.512875669 podStartE2EDuration="1.512875669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 16:17:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:18:00.511358499 +0000 UTC m=+2157.864877701" watchObservedRunningTime="2026-09-10 16:18:00.512875669 +0000 UTC m=+2157.866394831" Sep 10 16:18:01.225300 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:18:01.225274 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nt5ps_93cdfcd5-7e56-4acf-ac1b-eb9896ba6581/dns/0.log" Sep 10 16:18:01.252848 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:18:01.252816 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nt5ps_93cdfcd5-7e56-4acf-ac1b-eb9896ba6581/kube-rbac-proxy/0.log" Sep 10 16:18:01.588930 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:18:01.588894 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5zp9c_3b8c9295-101d-4700-a819-ae5172b0deee/dns-node-resolver/0.log" Sep 10 16:18:02.211576 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:18:02.211548 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xwmfs_322fb473-4c72-4194-9bba-9728743cdd74/node-ca/0.log" Sep 10 16:18:03.196665 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:18:03.196639 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz_df543b0d-1120-499b-8a4a-ea71ca5e539d/istio-proxy/0.log" Sep 10 16:18:03.623833 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:18:03.623800 2576 scope.go:117] "RemoveContainer" containerID="30992a809fe065a0309332095b5d51b389e4e0d3fbfc644e35f8d8a8b6024837" Sep 10 16:18:03.668348 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:18:03.668318 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-4xws7_eff52c89-238c-4887-b268-2bf69a3fabab/discovery/0.log" Sep 10 16:18:03.751957 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:18:03.751919 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-644bb569f8-dslcn_0b5650cc-b071-4c0a-a616-16012fb0a0a0/kube-auth-proxy/0.log" Sep 10 16:18:03.827664 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:18:03.827628 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-fq9cv_0b754538-f8b9-4891-aa4f-b910ddd9d35e/istio-proxy/0.log" Sep 10 16:18:03.843683 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:18:03.843648 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-fq9cv_0b754538-f8b9-4891-aa4f-b910ddd9d35e/istio-proxy/1.log" Sep 10 16:18:03.915644 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:18:03.915568 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-9g9rc_b5c2d260-e0f6-4e9d-a217-1c4ada9c1ade/payload-pre-processing/0.log" Sep 10 16:18:04.925645 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:18:04.925616 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8rhvj_9191d5ea-e8a5-4674-86cd-05d2c6949141/serve-healthcheck-canary/0.log" Sep 10 16:18:05.876602 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:18:05.876574 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wqh2d_0d22a485-8c50-428d-8da7-8ca4e831c2c8/kube-rbac-proxy/0.log" Sep 10 16:18:05.907207 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:18:05.907169 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wqh2d_0d22a485-8c50-428d-8da7-8ca4e831c2c8/exporter/0.log" Sep 10 16:18:05.940112 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:18:05.940087 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wqh2d_0d22a485-8c50-428d-8da7-8ca4e831c2c8/extractor/0.log" Sep 10 16:18:06.507837 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:18:06.507804 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-t86vd/perf-node-gather-daemonset-brwp4" Sep 10 16:18:08.521162 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:18:08.521126 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-n9wld_da14561b-2f8c-4987-953f-260f78e954db/manager/0.log" Sep 10 16:18:08.878331 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:18:08.878235 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-w7jnk_bfd89460-9cbb-4808-ad22-8b14f6e690ea/manager/0.log" Sep 10 16:18:08.955795 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:18:08.955761 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-pn66h_75f36c0a-f51b-4ee6-8917-f9153bfa9f2d/server/0.log" Sep 10 16:18:09.090993 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:18:09.090962 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-8h7gh_fc2d768d-cf98-4412-a63c-1ad1c466cdc5/manager/0.log" Sep 10 16:18:09.121851 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:18:09.121818 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-97f7948ff-cr5vg_899506ad-9415-4a67-b953-3d8134d64297/manager/0.log" Sep 10 16:18:09.147443 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:18:09.147368 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-97f7948ff-cr5vg_899506ad-9415-4a67-b953-3d8134d64297/copy-manifests/0.log" Sep 10 16:18:09.530271 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:18:09.530241 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz_df543b0d-1120-499b-8a4a-ea71ca5e539d/istio-proxy/0.log" Sep 10 16:18:09.984721 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:18:09.984685 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-4xws7_eff52c89-238c-4887-b268-2bf69a3fabab/discovery/0.log" Sep 10 16:18:10.014099 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:18:10.014069 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-644bb569f8-dslcn_0b5650cc-b071-4c0a-a616-16012fb0a0a0/kube-auth-proxy/0.log" Sep 10 16:18:10.034591 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:18:10.034563 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-fq9cv_0b754538-f8b9-4891-aa4f-b910ddd9d35e/istio-proxy/0.log" Sep 10 16:18:10.051274 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:18:10.051243 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-fq9cv_0b754538-f8b9-4891-aa4f-b910ddd9d35e/istio-proxy/1.log" Sep 10 16:18:10.089086 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:18:10.089052 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-9g9rc_b5c2d260-e0f6-4e9d-a217-1c4ada9c1ade/payload-pre-processing/0.log" Sep 10 16:18:10.722978 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:18:10.722932 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-pjmnk_fd5e7781-339a-418a-aa14-6b1ae3ff816a/openshift-lws-operator/0.log" Sep 10 16:18:11.689236 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:18:11.689197 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-c6qlz_df543b0d-1120-499b-8a4a-ea71ca5e539d/istio-proxy/0.log" Sep 10 16:18:12.153142 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:18:12.153107 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-4xws7_eff52c89-238c-4887-b268-2bf69a3fabab/discovery/0.log" Sep 10 16:18:12.205537 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:18:12.205502 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-644bb569f8-dslcn_0b5650cc-b071-4c0a-a616-16012fb0a0a0/kube-auth-proxy/0.log" Sep 10 16:18:12.234870 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:18:12.234841 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-fq9cv_0b754538-f8b9-4891-aa4f-b910ddd9d35e/istio-proxy/0.log" Sep 10 16:18:12.250042 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:18:12.250011 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-fq9cv_0b754538-f8b9-4891-aa4f-b910ddd9d35e/istio-proxy/1.log" Sep 10 16:18:12.299980 ip-10-0-139-236 kubenswrapper[2576]: I0910 16:18:12.299922 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-9g9rc_b5c2d260-e0f6-4e9d-a217-1c4ada9c1ade/payload-pre-processing/0.log"