Sep 12 01:43:11.201409 ip-10-0-133-79 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 12 01:43:11.201421 ip-10-0-133-79 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 12 01:43:11.201428 ip-10-0-133-79 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 12 01:43:11.201734 ip-10-0-133-79 systemd[1]: Failed to start Kubernetes Kubelet. Sep 12 01:43:20.893490 ip-10-0-133-79 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 12 01:43:20.893504 ip-10-0-133-79 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 4966dfed648c43a59bbaff3b6ed0dcf5 -- Sep 12 01:44:55.951208 ip-10-0-133-79 systemd[1]: Starting Kubernetes Kubelet... Sep 12 01:44:56.376913 ip-10-0-133-79 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 12 01:44:56.376913 ip-10-0-133-79 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 12 01:44:56.376913 ip-10-0-133-79 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 12 01:44:56.376913 ip-10-0-133-79 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 12 01:44:56.376913 ip-10-0-133-79 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 12 01:44:56.377609 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.376999 2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 12 01:44:56.385166 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385140 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 12 01:44:56.385166 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385158 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 12 01:44:56.385166 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385162 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 12 01:44:56.385166 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385165 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 12 01:44:56.385166 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385168 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 12 01:44:56.385166 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385171 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 12 01:44:56.385166 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385173 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 12 01:44:56.385166 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385176 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 12 01:44:56.385441 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385179 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 12 01:44:56.385441 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385182 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 12 01:44:56.385441 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385184 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 12 01:44:56.385441 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385189 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 12 01:44:56.385441 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385194 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 12 01:44:56.385441 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385197 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 12 01:44:56.385441 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385200 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 12 01:44:56.385441 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385202 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 12 01:44:56.385441 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385205 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 12 01:44:56.385441 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385208 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 12 01:44:56.385441 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385210 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 12 01:44:56.385441 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385213 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 12 01:44:56.385441 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385215 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 12 01:44:56.385441 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385218 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 12 01:44:56.385441 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385220 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 12 01:44:56.385441 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385223 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 12 01:44:56.385441 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385225 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 12 01:44:56.385441 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385228 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 12 01:44:56.385441 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385230 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 12 01:44:56.385916 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385233 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 12 01:44:56.385916 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385236 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 12 01:44:56.385916 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385238 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 12 01:44:56.385916 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385241 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 12 01:44:56.385916 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385243 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 12 01:44:56.385916 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385249 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 12 01:44:56.385916 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385251 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 12 01:44:56.385916 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385253 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 12 01:44:56.385916 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385256 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 12 01:44:56.385916 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385258 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 12 01:44:56.385916 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385261 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 12 01:44:56.385916 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385263 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 12 01:44:56.385916 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385266 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 12 01:44:56.385916 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385268 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 12 01:44:56.385916 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385272 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 12 01:44:56.385916 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385274 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 12 01:44:56.385916 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385277 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 12 01:44:56.385916 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385279 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 12 01:44:56.385916 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385282 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 12 01:44:56.385916 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385285 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 12 01:44:56.386411 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385288 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 12 01:44:56.386411 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385290 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 12 01:44:56.386411 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385294 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 12 01:44:56.386411 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385298 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 12 01:44:56.386411 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385301 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 12 01:44:56.386411 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385303 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 12 01:44:56.386411 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385306 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 12 01:44:56.386411 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385309 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 12 01:44:56.386411 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385312 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 12 01:44:56.386411 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385315 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 12 01:44:56.386411 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385317 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 12 01:44:56.386411 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385320 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 12 01:44:56.386411 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385322 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 12 01:44:56.386411 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385325 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 12 01:44:56.386411 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385327 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 12 01:44:56.386411 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385329 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 12 01:44:56.386411 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385332 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 12 01:44:56.386411 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385335 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 12 01:44:56.386411 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385339 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 12 01:44:56.386989 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385342 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 12 01:44:56.386989 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385345 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 12 01:44:56.386989 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385347 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 12 01:44:56.386989 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385350 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 12 01:44:56.386989 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385353 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 12 01:44:56.386989 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385355 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 12 01:44:56.386989 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385358 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 12 01:44:56.386989 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385360 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 12 01:44:56.386989 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385363 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 12 01:44:56.386989 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385365 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 12 01:44:56.386989 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385368 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 12 01:44:56.386989 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385370 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 12 01:44:56.386989 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385373 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 12 01:44:56.386989 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385375 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 12 01:44:56.386989 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385377 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 12 01:44:56.386989 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385381 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 12 01:44:56.386989 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385383 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 12 01:44:56.386989 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385386 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 12 01:44:56.386989 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385389 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 12 01:44:56.386989 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385391 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 12 01:44:56.387483 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385394 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 12 01:44:56.387483 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385396 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 12 01:44:56.387483 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385398 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 12 01:44:56.387483 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385401 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 12 01:44:56.387483 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385403 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 12 01:44:56.387483 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385405 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 12 01:44:56.387483 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385408 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 12 01:44:56.387483 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385410 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 12 01:44:56.387483 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385413 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 12 01:44:56.387483 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385416 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 12 01:44:56.387483 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385418 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 12 01:44:56.387483 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385420 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 12 01:44:56.387483 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.385423 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 12 01:44:56.387483 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386145 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 12 01:44:56.387483 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386152 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 12 01:44:56.387483 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386157 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 12 01:44:56.387483 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386160 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 12 01:44:56.387483 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386163 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 12 01:44:56.387483 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386166 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 12 01:44:56.387483 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386170 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 12 01:44:56.388012 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386173 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 12 01:44:56.388012 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386176 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 12 01:44:56.388012 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386179 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 12 01:44:56.388012 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386181 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 12 01:44:56.388012 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386184 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 12 01:44:56.388012 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386187 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 12 01:44:56.388012 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386189 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 12 01:44:56.388012 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386192 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 12 01:44:56.388012 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386194 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 12 01:44:56.388012 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386197 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 12 01:44:56.388012 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386200 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 12 01:44:56.388012 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386203 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 12 01:44:56.388012 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386206 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 12 01:44:56.388012 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386208 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 12 01:44:56.388012 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386211 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 12 01:44:56.388012 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386213 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 12 01:44:56.388012 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386216 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 12 01:44:56.388012 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386218 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 12 01:44:56.388012 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386221 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 12 01:44:56.388012 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386223 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 12 01:44:56.388503 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386226 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 12 01:44:56.388503 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386228 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 12 01:44:56.388503 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386231 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 12 01:44:56.388503 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386234 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 12 01:44:56.388503 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386236 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 12 01:44:56.388503 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386239 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 12 01:44:56.388503 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386241 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 12 01:44:56.388503 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386244 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 12 01:44:56.388503 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386248 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 12 01:44:56.388503 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386251 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 12 01:44:56.388503 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386254 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 12 01:44:56.388503 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386258 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 12 01:44:56.388503 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386261 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 12 01:44:56.388503 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386264 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 12 01:44:56.388503 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386266 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 12 01:44:56.388503 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386269 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 12 01:44:56.388503 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386271 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 12 01:44:56.388503 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386274 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 12 01:44:56.388503 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386276 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 12 01:44:56.388503 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386280 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 12 01:44:56.389015 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386282 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 12 01:44:56.389015 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386285 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 12 01:44:56.389015 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386287 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 12 01:44:56.389015 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386290 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 12 01:44:56.389015 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386293 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 12 01:44:56.389015 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386295 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 12 01:44:56.389015 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386298 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 12 01:44:56.389015 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386300 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 12 01:44:56.389015 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386303 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 12 01:44:56.389015 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386306 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 12 01:44:56.389015 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386308 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 12 01:44:56.389015 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386310 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 12 01:44:56.389015 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386313 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 12 01:44:56.389015 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386315 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 12 01:44:56.389015 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386317 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 12 01:44:56.389015 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386320 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 12 01:44:56.389015 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386323 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 12 01:44:56.389015 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386325 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 12 01:44:56.389015 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386328 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 12 01:44:56.389015 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386330 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 12 01:44:56.389621 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386333 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 12 01:44:56.389621 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386335 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 12 01:44:56.389621 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386338 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 12 01:44:56.389621 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386341 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 12 01:44:56.389621 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386343 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 12 01:44:56.389621 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386346 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 12 01:44:56.389621 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386348 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 12 01:44:56.389621 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386350 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 12 01:44:56.389621 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386353 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 12 01:44:56.389621 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386355 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 12 01:44:56.389621 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386357 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 12 01:44:56.389621 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386360 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 12 01:44:56.389621 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386362 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 12 01:44:56.389621 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386365 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 12 01:44:56.389621 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386367 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 12 01:44:56.389621 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386370 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 12 01:44:56.389621 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386372 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 12 01:44:56.389621 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386375 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 12 01:44:56.389621 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386377 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 12 01:44:56.389621 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386381 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 12 01:44:56.390155 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386384 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 12 01:44:56.390155 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386388 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 12 01:44:56.390155 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386390 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 12 01:44:56.390155 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386393 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 12 01:44:56.390155 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386395 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 12 01:44:56.390155 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386398 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 12 01:44:56.390155 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386400 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 12 01:44:56.390155 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386404 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 12 01:44:56.390155 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386407 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 12 01:44:56.390155 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386410 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 12 01:44:56.390155 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386413 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 12 01:44:56.390155 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386416 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 12 01:44:56.390155 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386819 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 12 01:44:56.390155 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386825 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 12 01:44:56.390155 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386828 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 12 01:44:56.390155 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386831 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 12 01:44:56.390155 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386835 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 12 01:44:56.390155 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386839 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 12 01:44:56.390155 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386843 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 12 01:44:56.390626 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386846 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 12 01:44:56.390626 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386849 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 12 01:44:56.390626 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386852 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 12 01:44:56.390626 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386855 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 12 01:44:56.390626 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386857 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 12 01:44:56.390626 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386860 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 12 01:44:56.390626 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386863 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 12 01:44:56.390626 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386866 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 12 01:44:56.390626 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386869 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 12 01:44:56.390626 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386875 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 12 01:44:56.390626 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386877 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 12 01:44:56.390626 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386880 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 12 01:44:56.390626 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386882 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 12 01:44:56.390626 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386885 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 12 01:44:56.390626 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386887 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 12 01:44:56.390626 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386890 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 12 01:44:56.390626 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386892 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 12 01:44:56.390626 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386895 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 12 01:44:56.390626 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386911 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 12 01:44:56.390626 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386914 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 12 01:44:56.391134 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386917 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 12 01:44:56.391134 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386921 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 12 01:44:56.391134 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386923 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 12 01:44:56.391134 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386926 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 12 01:44:56.391134 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386929 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 12 01:44:56.391134 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386932 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 12 01:44:56.391134 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386934 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 12 01:44:56.391134 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386937 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 12 01:44:56.391134 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386942 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 12 01:44:56.391134 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386947 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 12 01:44:56.391134 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386950 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 12 01:44:56.391134 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386953 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 12 01:44:56.391134 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386956 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 12 01:44:56.391134 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386959 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 12 01:44:56.391134 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386961 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 12 01:44:56.391134 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386964 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 12 01:44:56.391134 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386966 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 12 01:44:56.391134 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386969 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 12 01:44:56.391134 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386972 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 12 01:44:56.391625 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386974 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 12 01:44:56.391625 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386977 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 12 01:44:56.391625 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386979 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 12 01:44:56.391625 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386984 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 12 01:44:56.391625 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386987 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 12 01:44:56.391625 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386990 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 12 01:44:56.391625 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386993 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 12 01:44:56.391625 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386995 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 12 01:44:56.391625 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.386998 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 12 01:44:56.391625 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.387001 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 12 01:44:56.391625 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.387003 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 12 01:44:56.391625 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.387006 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 12 01:44:56.391625 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.387008 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 12 01:44:56.391625 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.387011 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 12 01:44:56.391625 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.387014 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 12 01:44:56.391625 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.387016 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 12 01:44:56.391625 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.387019 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 12 01:44:56.391625 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.387022 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 12 01:44:56.391625 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.387024 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 12 01:44:56.391625 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.387027 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 12 01:44:56.392128 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.387030 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 12 01:44:56.392128 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.387034 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 12 01:44:56.392128 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.387037 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 12 01:44:56.392128 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.387040 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 12 01:44:56.392128 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.387042 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 12 01:44:56.392128 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.387045 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 12 01:44:56.392128 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.387047 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 12 01:44:56.392128 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.387050 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 12 01:44:56.392128 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.387052 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 12 01:44:56.392128 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.387055 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 12 01:44:56.392128 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.387057 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 12 01:44:56.392128 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.387060 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 12 01:44:56.392128 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.387062 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 12 01:44:56.392128 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.387064 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 12 01:44:56.392128 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.387067 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 12 01:44:56.392128 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.387069 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 12 01:44:56.392128 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.387072 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 12 01:44:56.392128 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.387075 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 12 01:44:56.392128 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.387077 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 12 01:44:56.392128 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.387080 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 12 01:44:56.392611 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.387082 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 12 01:44:56.392611 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.387084 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 12 01:44:56.392611 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.387099 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 12 01:44:56.392611 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.387103 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 12 01:44:56.392611 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.387109 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 12 01:44:56.392611 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.387112 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 12 01:44:56.392611 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.387116 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 12 01:44:56.392611 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.387119 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 12 01:44:56.392611 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.387124 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 12 01:44:56.392611 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.387128 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 12 01:44:56.392611 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.387132 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 12 01:44:56.392611 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.387135 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 12 01:44:56.392611 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.387140 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 12 01:44:56.392611 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387231 2579 flags.go:64] FLAG: --address="0.0.0.0" Sep 12 01:44:56.392611 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387277 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 12 01:44:56.392611 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387287 2579 flags.go:64] FLAG: --anonymous-auth="true" Sep 12 01:44:56.392611 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387294 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 12 01:44:56.392611 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387301 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 12 01:44:56.392611 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387305 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 12 01:44:56.392611 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387312 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 12 01:44:56.392611 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387318 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 12 01:44:56.393173 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387323 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 12 01:44:56.393173 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387328 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 12 01:44:56.393173 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387333 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 12 01:44:56.393173 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387338 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 12 01:44:56.393173 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387343 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 12 01:44:56.393173 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387348 2579 flags.go:64] FLAG: --cgroup-root="" Sep 12 01:44:56.393173 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387352 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 12 01:44:56.393173 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387357 2579 flags.go:64] FLAG: --client-ca-file="" Sep 12 01:44:56.393173 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387362 2579 flags.go:64] FLAG: --cloud-provider="external" Sep 12 01:44:56.393173 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387367 2579 flags.go:64] FLAG: --cluster-dns="[]" Sep 12 01:44:56.393173 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387373 2579 flags.go:64] FLAG: --cluster-domain="" Sep 12 01:44:56.393173 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387377 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 12 01:44:56.393173 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387382 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 12 01:44:56.393173 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387386 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 12 01:44:56.393173 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387392 2579 flags.go:64] FLAG: --container-log-max-files="5" Sep 12 01:44:56.393173 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387401 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 12 01:44:56.393173 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387406 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 12 01:44:56.393173 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387411 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 12 01:44:56.393173 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387417 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 12 01:44:56.393173 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387421 2579 flags.go:64] FLAG: --contention-profiling="false" Sep 12 01:44:56.393173 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387426 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 12 01:44:56.393173 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387431 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 12 01:44:56.393173 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387435 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 12 01:44:56.393173 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387439 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 12 01:44:56.393751 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387448 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 12 01:44:56.393751 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387454 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 12 01:44:56.393751 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387458 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 12 01:44:56.393751 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387463 2579 flags.go:64] FLAG: --enable-load-reader="false" Sep 12 01:44:56.393751 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387467 2579 flags.go:64] FLAG: --enable-server="true" Sep 12 01:44:56.393751 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387472 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 12 01:44:56.393751 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387478 2579 flags.go:64] FLAG: --event-burst="100" Sep 12 01:44:56.393751 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387483 2579 flags.go:64] FLAG: --event-qps="50" Sep 12 01:44:56.393751 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387487 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 12 01:44:56.393751 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387493 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 12 01:44:56.393751 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387497 2579 flags.go:64] FLAG: --eviction-hard="" Sep 12 01:44:56.393751 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387503 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 12 01:44:56.393751 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387507 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 12 01:44:56.393751 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387511 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 12 01:44:56.393751 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387516 2579 flags.go:64] FLAG: --eviction-soft="" Sep 12 01:44:56.393751 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387520 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 12 01:44:56.393751 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387525 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 12 01:44:56.393751 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387531 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 12 01:44:56.393751 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387536 2579 flags.go:64] FLAG: --experimental-mounter-path="" Sep 12 01:44:56.393751 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387541 2579 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 12 01:44:56.393751 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387545 2579 flags.go:64] FLAG: --fail-swap-on="true" Sep 12 01:44:56.393751 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387549 2579 flags.go:64] FLAG: --feature-gates="" Sep 12 01:44:56.393751 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387556 2579 flags.go:64] FLAG: --file-check-frequency="20s" Sep 12 01:44:56.393751 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387563 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 12 01:44:56.393751 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387569 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 12 01:44:56.394405 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387574 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 12 01:44:56.394405 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387580 2579 flags.go:64] FLAG: --healthz-port="10248" Sep 12 01:44:56.394405 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387585 2579 flags.go:64] FLAG: --help="false" Sep 12 01:44:56.394405 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387590 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-133-79.ec2.internal" Sep 12 01:44:56.394405 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387594 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 12 01:44:56.394405 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387598 2579 flags.go:64] FLAG: --http-check-frequency="20s" Sep 12 01:44:56.394405 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387602 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 12 01:44:56.394405 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387608 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 12 01:44:56.394405 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387612 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 12 01:44:56.394405 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387616 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 12 01:44:56.394405 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387620 2579 flags.go:64] FLAG: --image-service-endpoint="" Sep 12 01:44:56.394405 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387624 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 12 01:44:56.394405 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387628 2579 flags.go:64] FLAG: --kube-api-burst="100" Sep 12 01:44:56.394405 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387632 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 12 01:44:56.394405 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387637 2579 flags.go:64] FLAG: --kube-api-qps="50" Sep 12 01:44:56.394405 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387640 2579 flags.go:64] FLAG: --kube-reserved="" Sep 12 01:44:56.394405 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387645 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 12 01:44:56.394405 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387648 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 12 01:44:56.394405 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387653 2579 flags.go:64] FLAG: --kubelet-cgroups="" Sep 12 01:44:56.394405 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387657 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 12 01:44:56.394405 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387661 2579 flags.go:64] FLAG: --lock-file="" Sep 12 01:44:56.394405 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387666 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 12 01:44:56.394405 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387670 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 12 01:44:56.394405 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387674 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 12 01:44:56.395025 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387681 2579 flags.go:64] FLAG: --log-json-split-stream="false" Sep 12 01:44:56.395025 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387685 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 12 01:44:56.395025 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387690 2579 flags.go:64] FLAG: --log-text-split-stream="false" Sep 12 01:44:56.395025 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387693 2579 flags.go:64] FLAG: --logging-format="text" Sep 12 01:44:56.395025 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387697 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 12 01:44:56.395025 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387702 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 12 01:44:56.395025 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387707 2579 flags.go:64] FLAG: --manifest-url="" Sep 12 01:44:56.395025 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387712 2579 flags.go:64] FLAG: --manifest-url-header="" Sep 12 01:44:56.395025 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387718 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 12 01:44:56.395025 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387724 2579 flags.go:64] FLAG: --max-open-files="1000000" Sep 12 01:44:56.395025 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387729 2579 flags.go:64] FLAG: --max-pods="110" Sep 12 01:44:56.395025 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387734 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 12 01:44:56.395025 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387739 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 12 01:44:56.395025 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387743 2579 flags.go:64] FLAG: --memory-manager-policy="None" Sep 12 01:44:56.395025 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387747 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 12 01:44:56.395025 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387752 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 12 01:44:56.395025 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387757 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 12 01:44:56.395025 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387762 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 12 01:44:56.395025 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387779 2579 flags.go:64] FLAG: --node-status-max-images="50" Sep 12 01:44:56.395025 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387784 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 12 01:44:56.395025 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387788 2579 flags.go:64] FLAG: --oom-score-adj="-999" Sep 12 01:44:56.395025 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387793 2579 flags.go:64] FLAG: --pod-cidr="" Sep 12 01:44:56.395025 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387799 2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 12 01:44:56.395592 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387807 2579 flags.go:64] FLAG: --pod-manifest-path="" Sep 12 01:44:56.395592 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387812 2579 flags.go:64] FLAG: --pod-max-pids="-1" Sep 12 01:44:56.395592 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387817 2579 flags.go:64] FLAG: --pods-per-core="0" Sep 12 01:44:56.395592 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387821 2579 flags.go:64] FLAG: --port="10250" Sep 12 01:44:56.395592 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387826 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 12 01:44:56.395592 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387831 2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03770cee62ad9be9c" Sep 12 01:44:56.395592 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387836 2579 flags.go:64] FLAG: --qos-reserved="" Sep 12 01:44:56.395592 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387841 2579 flags.go:64] FLAG: --read-only-port="10255" Sep 12 01:44:56.395592 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387846 2579 flags.go:64] FLAG: --register-node="true" Sep 12 01:44:56.395592 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387853 2579 flags.go:64] FLAG: --register-with-taints="" Sep 12 01:44:56.395592 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387859 2579 flags.go:64] FLAG: --registry-burst="10" Sep 12 01:44:56.395592 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387864 2579 flags.go:64] FLAG: --registry-qps="5" Sep 12 01:44:56.395592 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387869 2579 flags.go:64] FLAG: --reserved-cpus="" Sep 12 01:44:56.395592 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387873 2579 flags.go:64] FLAG: --reserved-memory="" Sep 12 01:44:56.395592 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387879 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 12 01:44:56.395592 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387884 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 12 01:44:56.395592 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387890 2579 flags.go:64] FLAG: --rotate-certificates="false" Sep 12 01:44:56.395592 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387895 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 12 01:44:56.395592 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387921 2579 flags.go:64] FLAG: --runonce="false" Sep 12 01:44:56.395592 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387926 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 12 01:44:56.395592 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387931 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 12 01:44:56.395592 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387936 2579 flags.go:64] FLAG: --seccomp-default="false" Sep 12 01:44:56.395592 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387941 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 12 01:44:56.395592 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387945 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 12 01:44:56.395592 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387950 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 12 01:44:56.395592 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387954 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 12 01:44:56.396285 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387959 2579 flags.go:64] FLAG: --storage-driver-password="root" Sep 12 01:44:56.396285 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387964 2579 flags.go:64] FLAG: --storage-driver-secure="false" Sep 12 01:44:56.396285 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387969 2579 flags.go:64] FLAG: --storage-driver-table="stats" Sep 12 01:44:56.396285 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.387974 2579 flags.go:64] FLAG: --storage-driver-user="root" Sep 12 01:44:56.396285 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.388005 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 12 01:44:56.396285 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.388013 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 12 01:44:56.396285 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.388019 2579 flags.go:64] FLAG: --system-cgroups="" Sep 12 01:44:56.396285 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.388025 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 12 01:44:56.396285 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.388035 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 12 01:44:56.396285 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.388040 2579 flags.go:64] FLAG: --tls-cert-file="" Sep 12 01:44:56.396285 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.388045 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 12 01:44:56.396285 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.388052 2579 flags.go:64] FLAG: --tls-min-version="" Sep 12 01:44:56.396285 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.388057 2579 flags.go:64] FLAG: --tls-private-key-file="" Sep 12 01:44:56.396285 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.388062 2579 flags.go:64] FLAG: --topology-manager-policy="none" Sep 12 01:44:56.396285 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.388066 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 12 01:44:56.396285 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.388071 2579 flags.go:64] FLAG: --topology-manager-scope="container" Sep 12 01:44:56.396285 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.388076 2579 flags.go:64] FLAG: --v="2" Sep 12 01:44:56.396285 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.388083 2579 flags.go:64] FLAG: --version="false" Sep 12 01:44:56.396285 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.388090 2579 flags.go:64] FLAG: --vmodule="" Sep 12 01:44:56.396285 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.388096 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 12 01:44:56.396285 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.388102 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 12 01:44:56.396285 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388293 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 12 01:44:56.396285 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388302 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 12 01:44:56.396285 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388308 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 12 01:44:56.396285 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388313 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 12 01:44:56.396960 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388317 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 12 01:44:56.396960 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388322 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 12 01:44:56.396960 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388326 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 12 01:44:56.396960 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388331 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 12 01:44:56.396960 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388335 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 12 01:44:56.396960 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388340 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 12 01:44:56.396960 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388345 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 12 01:44:56.396960 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388349 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 12 01:44:56.396960 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388353 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 12 01:44:56.396960 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388358 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 12 01:44:56.396960 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388362 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 12 01:44:56.396960 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388367 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 12 01:44:56.396960 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388371 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 12 01:44:56.396960 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388375 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 12 01:44:56.396960 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388379 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 12 01:44:56.396960 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388383 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 12 01:44:56.396960 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388388 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 12 01:44:56.396960 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388392 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 12 01:44:56.396960 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388396 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 12 01:44:56.396960 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388401 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 12 01:44:56.397460 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388405 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 12 01:44:56.397460 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388410 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 12 01:44:56.397460 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388414 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 12 01:44:56.397460 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388418 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 12 01:44:56.397460 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388422 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 12 01:44:56.397460 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388426 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 12 01:44:56.397460 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388431 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 12 01:44:56.397460 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388436 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 12 01:44:56.397460 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388440 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 12 01:44:56.397460 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388445 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 12 01:44:56.397460 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388449 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 12 01:44:56.397460 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388453 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 12 01:44:56.397460 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388457 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 12 01:44:56.397460 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388461 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 12 01:44:56.397460 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388465 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 12 01:44:56.397460 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388469 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 12 01:44:56.397460 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388473 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 12 01:44:56.397460 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388479 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 12 01:44:56.397460 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388483 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 12 01:44:56.397460 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388488 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 12 01:44:56.397961 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388492 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 12 01:44:56.397961 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388496 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 12 01:44:56.397961 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388500 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 12 01:44:56.397961 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388504 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 12 01:44:56.397961 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388509 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 12 01:44:56.397961 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388513 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 12 01:44:56.397961 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388518 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 12 01:44:56.397961 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388523 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 12 01:44:56.397961 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388526 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 12 01:44:56.397961 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388533 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 12 01:44:56.397961 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388539 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 12 01:44:56.397961 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388543 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 12 01:44:56.397961 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388548 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 12 01:44:56.397961 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388553 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 12 01:44:56.397961 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388558 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 12 01:44:56.397961 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388562 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 12 01:44:56.397961 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388567 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 12 01:44:56.397961 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388572 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 12 01:44:56.397961 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388577 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 12 01:44:56.398429 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388581 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 12 01:44:56.398429 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388586 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 12 01:44:56.398429 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388592 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 12 01:44:56.398429 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388596 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 12 01:44:56.398429 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388602 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 12 01:44:56.398429 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388607 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 12 01:44:56.398429 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388612 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 12 01:44:56.398429 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388616 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 12 01:44:56.398429 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388621 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 12 01:44:56.398429 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388625 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 12 01:44:56.398429 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388631 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 12 01:44:56.398429 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388636 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 12 01:44:56.398429 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388642 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 12 01:44:56.398429 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388649 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 12 01:44:56.398429 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388654 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 12 01:44:56.398429 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388658 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 12 01:44:56.398429 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388662 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 12 01:44:56.398429 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388666 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 12 01:44:56.398429 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388671 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 12 01:44:56.398896 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388675 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 12 01:44:56.398896 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388682 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 12 01:44:56.398896 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388687 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 12 01:44:56.398896 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388692 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 12 01:44:56.398896 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388697 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 12 01:44:56.398896 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388701 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 12 01:44:56.398896 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388706 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 12 01:44:56.398896 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388711 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 12 01:44:56.398896 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388715 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 12 01:44:56.398896 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388719 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 12 01:44:56.398896 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388723 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 12 01:44:56.398896 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388728 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 12 01:44:56.398896 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388732 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 12 01:44:56.398896 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388736 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 12 01:44:56.398896 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388742 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 12 01:44:56.398896 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388747 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 12 01:44:56.398896 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.388751 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 12 01:44:56.399331 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.389716 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 12 01:44:56.399331 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.397561 2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 12 01:44:56.399331 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.397578 2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 12 01:44:56.399331 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.397598 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 12 01:44:56.399331 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.397604 2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 12 01:44:56.399331 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397662 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 12 01:44:56.399331 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397667 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 12 01:44:56.399331 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397670 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 12 01:44:56.399331 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397673 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 12 01:44:56.399331 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397676 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 12 01:44:56.399331 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397679 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 12 01:44:56.399331 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397683 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 12 01:44:56.399331 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397687 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 12 01:44:56.399331 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397690 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 12 01:44:56.399679 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397693 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 12 01:44:56.399679 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397696 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 12 01:44:56.399679 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397699 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 12 01:44:56.399679 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397701 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 12 01:44:56.399679 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397704 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 12 01:44:56.399679 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397707 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 12 01:44:56.399679 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397709 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 12 01:44:56.399679 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397712 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 12 01:44:56.399679 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397714 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 12 01:44:56.399679 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397716 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 12 01:44:56.399679 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397719 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 12 01:44:56.399679 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397723 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 12 01:44:56.399679 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397727 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 12 01:44:56.399679 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397730 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 12 01:44:56.399679 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397732 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 12 01:44:56.399679 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397735 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 12 01:44:56.399679 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397738 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 12 01:44:56.399679 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397740 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 12 01:44:56.399679 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397743 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 12 01:44:56.399679 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397746 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 12 01:44:56.400218 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397750 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 12 01:44:56.400218 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397753 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 12 01:44:56.400218 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397756 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 12 01:44:56.400218 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397758 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 12 01:44:56.400218 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397761 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 12 01:44:56.400218 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397763 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 12 01:44:56.400218 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397766 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 12 01:44:56.400218 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397768 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 12 01:44:56.400218 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397771 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 12 01:44:56.400218 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397773 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 12 01:44:56.400218 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397776 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 12 01:44:56.400218 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397779 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 12 01:44:56.400218 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397781 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 12 01:44:56.400218 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397783 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 12 01:44:56.400218 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397786 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 12 01:44:56.400218 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397788 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 12 01:44:56.400218 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397791 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 12 01:44:56.400218 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397793 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 12 01:44:56.400218 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397796 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 12 01:44:56.400679 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397798 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 12 01:44:56.400679 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397800 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 12 01:44:56.400679 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397803 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 12 01:44:56.400679 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397805 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 12 01:44:56.400679 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397808 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 12 01:44:56.400679 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397810 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 12 01:44:56.400679 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397812 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 12 01:44:56.400679 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397828 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 12 01:44:56.400679 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397830 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 12 01:44:56.400679 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397833 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 12 01:44:56.400679 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397835 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 12 01:44:56.400679 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397838 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 12 01:44:56.400679 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397841 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 12 01:44:56.400679 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397844 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 12 01:44:56.400679 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397847 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 12 01:44:56.400679 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397850 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 12 01:44:56.400679 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397852 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 12 01:44:56.400679 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397855 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 12 01:44:56.400679 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397857 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 12 01:44:56.400679 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397860 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 12 01:44:56.401279 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397863 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 12 01:44:56.401279 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397865 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 12 01:44:56.401279 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397868 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 12 01:44:56.401279 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397871 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 12 01:44:56.401279 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397873 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 12 01:44:56.401279 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397876 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 12 01:44:56.401279 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397878 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 12 01:44:56.401279 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397880 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 12 01:44:56.401279 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397883 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 12 01:44:56.401279 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397887 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 12 01:44:56.401279 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397890 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 12 01:44:56.401279 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397893 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 12 01:44:56.401279 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397896 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 12 01:44:56.401279 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397912 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 12 01:44:56.401279 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397914 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 12 01:44:56.401279 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397917 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 12 01:44:56.401279 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397919 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 12 01:44:56.401279 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397922 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 12 01:44:56.401279 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397925 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 12 01:44:56.401279 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397927 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 12 01:44:56.401786 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397929 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 12 01:44:56.401786 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397932 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 12 01:44:56.401786 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397935 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 12 01:44:56.401786 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397937 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 12 01:44:56.401786 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397940 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 12 01:44:56.401786 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397943 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 12 01:44:56.401786 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397946 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 12 01:44:56.401786 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397949 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 12 01:44:56.401786 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397952 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 12 01:44:56.401786 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397954 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 12 01:44:56.401786 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.397957 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 12 01:44:56.401786 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.397962 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 12 01:44:56.401786 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398079 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 12 01:44:56.401786 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398084 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 12 01:44:56.401786 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398087 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 12 01:44:56.402190 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398090 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 12 01:44:56.402190 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398093 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 12 01:44:56.402190 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398096 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 12 01:44:56.402190 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398098 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 12 01:44:56.402190 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398101 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 12 01:44:56.402190 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398103 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 12 01:44:56.402190 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398105 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 12 01:44:56.402190 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398108 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 12 01:44:56.402190 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398110 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 12 01:44:56.402190 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398113 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 12 01:44:56.402190 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398116 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 12 01:44:56.402190 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398118 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 12 01:44:56.402190 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398121 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 12 01:44:56.402190 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398123 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 12 01:44:56.402190 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398125 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 12 01:44:56.402190 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398128 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 12 01:44:56.402190 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398130 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 12 01:44:56.402190 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398133 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 12 01:44:56.402190 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398135 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 12 01:44:56.402190 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398137 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 12 01:44:56.402686 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398140 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 12 01:44:56.402686 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398142 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 12 01:44:56.402686 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398146 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 12 01:44:56.402686 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398150 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 12 01:44:56.402686 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398154 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 12 01:44:56.402686 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398157 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 12 01:44:56.402686 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398160 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 12 01:44:56.402686 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398163 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 12 01:44:56.402686 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398166 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 12 01:44:56.402686 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398169 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 12 01:44:56.402686 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398171 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 12 01:44:56.402686 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398174 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 12 01:44:56.402686 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398177 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 12 01:44:56.402686 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398179 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 12 01:44:56.402686 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398181 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 12 01:44:56.402686 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398184 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 12 01:44:56.402686 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398186 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 12 01:44:56.402686 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398189 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 12 01:44:56.402686 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398191 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 12 01:44:56.403170 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398194 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 12 01:44:56.403170 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398196 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 12 01:44:56.403170 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398199 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 12 01:44:56.403170 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398201 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 12 01:44:56.403170 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398204 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 12 01:44:56.403170 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398206 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 12 01:44:56.403170 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398210 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 12 01:44:56.403170 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398213 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 12 01:44:56.403170 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398216 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 12 01:44:56.403170 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398220 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 12 01:44:56.403170 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398223 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 12 01:44:56.403170 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398225 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 12 01:44:56.403170 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398228 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 12 01:44:56.403170 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398231 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 12 01:44:56.403170 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398234 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 12 01:44:56.403170 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398236 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 12 01:44:56.403170 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398240 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 12 01:44:56.403170 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398242 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 12 01:44:56.403170 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398245 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 12 01:44:56.403617 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398248 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 12 01:44:56.403617 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398250 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 12 01:44:56.403617 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398253 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 12 01:44:56.403617 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398255 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 12 01:44:56.403617 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398258 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 12 01:44:56.403617 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398260 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 12 01:44:56.403617 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398263 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 12 01:44:56.403617 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398265 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 12 01:44:56.403617 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398268 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 12 01:44:56.403617 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398270 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 12 01:44:56.403617 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398273 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 12 01:44:56.403617 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398275 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 12 01:44:56.403617 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398277 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 12 01:44:56.403617 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398280 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 12 01:44:56.403617 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398283 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 12 01:44:56.403617 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398285 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 12 01:44:56.403617 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398288 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 12 01:44:56.403617 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398290 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 12 01:44:56.403617 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398293 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 12 01:44:56.403617 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398295 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 12 01:44:56.404104 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398298 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 12 01:44:56.404104 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398300 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 12 01:44:56.404104 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398303 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 12 01:44:56.404104 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398305 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 12 01:44:56.404104 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398308 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 12 01:44:56.404104 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398310 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 12 01:44:56.404104 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398312 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 12 01:44:56.404104 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398315 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 12 01:44:56.404104 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398318 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 12 01:44:56.404104 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398320 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 12 01:44:56.404104 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398323 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 12 01:44:56.404104 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398325 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 12 01:44:56.404104 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398328 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 12 01:44:56.404104 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398331 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 12 01:44:56.404104 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398334 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 12 01:44:56.404104 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398336 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 12 01:44:56.404104 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398339 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 12 01:44:56.404104 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:56.398341 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 12 01:44:56.404563 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.398346 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 12 01:44:56.404563 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.398457 2579 server.go:962] "Client rotation is on, will bootstrap in background" Sep 12 01:44:56.404563 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.401594 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 12 01:44:56.404563 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.402687 2579 server.go:1019] "Starting client certificate rotation" Sep 12 01:44:56.404563 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.402780 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 12 01:44:56.404563 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.403539 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 12 01:44:56.405607 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.405593 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 12 01:44:56.410496 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.410456 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 12 01:44:56.423014 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.422992 2579 log.go:25] "Validated CRI v1 runtime API" Sep 12 01:44:56.426513 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.426495 2579 log.go:25] "Validated CRI v1 image API" Sep 12 01:44:56.427403 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.427387 2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 12 01:44:56.430816 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.430794 2579 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 c4a100eb-ea94-48a9-a87e-4762dcee7c30:/dev/nvme0n1p3 ecc6f07b-a645-4ece-b783-68c893550a65:/dev/nvme0n1p4] Sep 12 01:44:56.430871 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.430816 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 12 01:44:56.435246 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.435123 2579 manager.go:216] Machine: {Timestamp:2026-09-12 01:44:56.43458812 +0000 UTC m=+0.368131987 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:2499996 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e9dbc35e8831a58f1ff266b8de8c0 SystemUUID:ec2e9dbc-35e8-831a-58f1-ff266b8de8c0 BootID:4966dfed-648c-43a5-9bba-ff3b6ed0dcf5 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ae:33:d6:b1:19 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ae:33:d6:b1:19 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:0a:ec:d8:cf:39:43 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 12 01:44:56.435246 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.435232 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 12 01:44:56.435371 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.435343 2579 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 12 01:44:56.435418 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.435398 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 12 01:44:56.437106 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.437080 2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 12 01:44:56.437267 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.437108 2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-79.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 12 01:44:56.437967 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.437957 2579 topology_manager.go:138] "Creating topology manager with none policy" Sep 12 01:44:56.438004 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.437970 2579 container_manager_linux.go:309] "Creating device plugin manager" Sep 12 01:44:56.438004 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.437978 2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 12 01:44:56.438934 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.438923 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 12 01:44:56.440712 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.440702 2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 12 01:44:56.444987 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.444974 2579 state_mem.go:36] "Initialized new in-memory state store" Sep 12 01:44:56.445142 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.445131 2579 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 12 01:44:56.445182 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.445165 2579 kubelet.go:486] "Attempting to sync node with API server" Sep 12 01:44:56.445210 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.445182 2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 12 01:44:56.445210 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.445196 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 12 01:44:56.445210 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.445206 2579 kubelet.go:398] "Adding apiserver pod source" Sep 12 01:44:56.445285 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.445214 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 12 01:44:56.445612 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.445599 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 12 01:44:56.445650 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.445624 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 12 01:44:56.448170 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.448151 2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 12 01:44:56.451139 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.451125 2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 12 01:44:56.451206 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.451156 2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 12 01:44:56.453095 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.453083 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 12 01:44:56.453142 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.453103 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 12 01:44:56.453142 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.453113 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 12 01:44:56.453142 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.453123 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 12 01:44:56.453142 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.453138 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 12 01:44:56.453251 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.453146 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 12 01:44:56.453251 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.453161 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 12 01:44:56.453251 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.453168 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 12 01:44:56.453251 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.453177 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 12 01:44:56.453251 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.453182 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 12 01:44:56.453251 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.453191 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 12 01:44:56.453251 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.453196 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 12 01:44:56.454009 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.453997 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 12 01:44:56.454041 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.454012 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 12 01:44:56.459918 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.459687 2579 server.go:1268] "Started kubelet" Sep 12 01:44:56.459997 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.459786 2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 12 01:44:56.460065 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.459776 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 12 01:44:56.460135 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.460114 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 12 01:44:56.460814 ip-10-0-133-79 systemd[1]: Started Kubernetes Kubelet. Sep 12 01:44:56.461675 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.461534 2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 12 01:44:56.462228 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.462218 2579 server.go:310] "Adding debug handlers to kubelet server" Sep 12 01:44:56.464098 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:44:56.464078 2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 12 01:44:56.467844 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.467829 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qwzfp" Sep 12 01:44:56.468336 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.468319 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 12 01:44:56.468431 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.468318 2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 12 01:44:56.468512 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:44:56.468445 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-79.ec2.internal\" not found" Sep 12 01:44:56.468598 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.468579 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-79.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 12 01:44:56.469144 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.469126 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 12 01:44:56.469144 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.469128 2579 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 12 01:44:56.469290 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.469157 2579 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 12 01:44:56.469290 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.469260 2579 reconstruct.go:97] "Volume reconstruction finished" Sep 12 01:44:56.469290 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.469273 2579 reconciler.go:29] "Reconciler: start to sync state" Sep 12 01:44:56.469429 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.469398 2579 factory.go:55] Registering systemd factory Sep 12 01:44:56.469429 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.469411 2579 factory.go:223] Registration of the systemd container factory successfully Sep 12 01:44:56.469646 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.469630 2579 factory.go:153] Registering CRI-O factory Sep 12 01:44:56.469720 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.469651 2579 factory.go:223] Registration of the crio container factory successfully Sep 12 01:44:56.469720 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.469715 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 12 01:44:56.469818 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.469732 2579 factory.go:103] Registering Raw factory Sep 12 01:44:56.469818 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.469754 2579 manager.go:1196] Started watching for new ooms in manager Sep 12 01:44:56.469979 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:44:56.469945 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 12 01:44:56.470083 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:44:56.468896 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-79.ec2.internal.18d46f741c998a60 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-79.ec2.internal,UID:ip-10-0-133-79.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-79.ec2.internal,},FirstTimestamp:2026-09-12 01:44:56.459643488 +0000 UTC m=+0.393187354,LastTimestamp:2026-09-12 01:44:56.459643488 +0000 UTC m=+0.393187354,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-79.ec2.internal,}" Sep 12 01:44:56.470613 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:44:56.470593 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-79.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 12 01:44:56.470921 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.470889 2579 manager.go:349] Starting recovery of all containers Sep 12 01:44:56.473916 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:44:56.473870 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 12 01:44:56.474111 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:44:56.474053 2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-79.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 12 01:44:56.477243 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.477153 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 12 01:44:56.479881 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.479856 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qwzfp" Sep 12 01:44:56.482526 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.482504 2579 manager.go:354] Recovery completed Sep 12 01:44:56.490158 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.490142 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 12 01:44:56.492120 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.492104 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-79.ec2.internal" event="NodeHasSufficientMemory" Sep 12 01:44:56.492180 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.492132 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-79.ec2.internal" event="NodeHasNoDiskPressure" Sep 12 01:44:56.492180 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.492144 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-79.ec2.internal" event="NodeHasSufficientPID" Sep 12 01:44:56.492703 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.492687 2579 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 12 01:44:56.492755 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.492705 2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 12 01:44:56.492755 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.492692 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 12 01:44:56.492755 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.492723 2579 state_mem.go:36] "Initialized new in-memory state store" Sep 12 01:44:56.494404 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:44:56.494337 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-79.ec2.internal.18d46f741e8915c6 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-79.ec2.internal,UID:ip-10-0-133-79.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-79.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-79.ec2.internal,},FirstTimestamp:2026-09-12 01:44:56.492119494 +0000 UTC m=+0.425663360,LastTimestamp:2026-09-12 01:44:56.492119494 +0000 UTC m=+0.425663360,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-79.ec2.internal,}" Sep 12 01:44:56.494533 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.494514 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-79.ec2.internal" event="NodeHasSufficientMemory" Sep 12 01:44:56.494622 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.494552 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-79.ec2.internal" event="NodeHasNoDiskPressure" Sep 12 01:44:56.494622 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.494567 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-79.ec2.internal" event="NodeHasSufficientPID" Sep 12 01:44:56.494752 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.494741 2579 policy_none.go:49] "None policy: Start" Sep 12 01:44:56.494798 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.494758 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 12 01:44:56.494798 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.494768 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 12 01:44:56.496915 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.496892 2579 policy_none.go:47] "Start" Sep 12 01:44:56.555739 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.533587 2579 manager.go:337] "Starting Device Plugin manager" Sep 12 01:44:56.555739 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:44:56.533621 2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 12 01:44:56.555739 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.533634 2579 server.go:87] "Starting device plugin registration server" Sep 12 01:44:56.555739 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.534009 2579 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 12 01:44:56.555739 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.534021 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 12 01:44:56.555739 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.534161 2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 12 01:44:56.555739 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.534242 2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 12 01:44:56.555739 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.534252 2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 12 01:44:56.555739 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:44:56.535009 2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 12 01:44:56.555739 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:44:56.535058 2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-79.ec2.internal\" not found" Sep 12 01:44:56.577814 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.577790 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 12 01:44:56.577959 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.577846 2579 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 12 01:44:56.577959 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.577868 2579 kubelet.go:2448] "Starting kubelet main sync loop" Sep 12 01:44:56.577959 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:44:56.577915 2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 12 01:44:56.580072 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.580054 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 12 01:44:56.635252 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.635120 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 12 01:44:56.636174 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.636152 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-79.ec2.internal" event="NodeHasSufficientMemory" Sep 12 01:44:56.636292 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.636189 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-79.ec2.internal" event="NodeHasNoDiskPressure" Sep 12 01:44:56.636292 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.636203 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-79.ec2.internal" event="NodeHasSufficientPID" Sep 12 01:44:56.636292 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.636233 2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-79.ec2.internal" Sep 12 01:44:56.644167 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.644151 2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-79.ec2.internal" Sep 12 01:44:56.644211 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:44:56.644174 2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-79.ec2.internal\": node \"ip-10-0-133-79.ec2.internal\" not found" Sep 12 01:44:56.657307 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:44:56.657285 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-79.ec2.internal\" not found" Sep 12 01:44:56.678948 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.678928 2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-79.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-79.ec2.internal"] Sep 12 01:44:56.679014 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.678988 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 12 01:44:56.679832 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.679815 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-79.ec2.internal" event="NodeHasSufficientMemory" Sep 12 01:44:56.679924 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.679848 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-79.ec2.internal" event="NodeHasNoDiskPressure" Sep 12 01:44:56.679924 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.679858 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-79.ec2.internal" event="NodeHasSufficientPID" Sep 12 01:44:56.682035 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.682021 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 12 01:44:56.682157 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.682139 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-79.ec2.internal" Sep 12 01:44:56.682197 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.682167 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 12 01:44:56.682729 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.682714 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-79.ec2.internal" event="NodeHasSufficientMemory" Sep 12 01:44:56.682809 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.682730 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-79.ec2.internal" event="NodeHasSufficientMemory" Sep 12 01:44:56.682809 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.682744 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-79.ec2.internal" event="NodeHasNoDiskPressure" Sep 12 01:44:56.682809 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.682753 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-79.ec2.internal" event="NodeHasNoDiskPressure" Sep 12 01:44:56.682809 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.682757 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-79.ec2.internal" event="NodeHasSufficientPID" Sep 12 01:44:56.682809 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.682763 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-79.ec2.internal" event="NodeHasSufficientPID" Sep 12 01:44:56.684749 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.684732 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-79.ec2.internal" Sep 12 01:44:56.684826 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.684764 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 12 01:44:56.685450 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.685434 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-79.ec2.internal" event="NodeHasSufficientMemory" Sep 12 01:44:56.685556 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.685465 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-79.ec2.internal" event="NodeHasNoDiskPressure" Sep 12 01:44:56.685556 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.685478 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-79.ec2.internal" event="NodeHasSufficientPID" Sep 12 01:44:56.704517 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:44:56.704496 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-79.ec2.internal\" not found" node="ip-10-0-133-79.ec2.internal" Sep 12 01:44:56.708799 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:44:56.708777 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-79.ec2.internal\" not found" node="ip-10-0-133-79.ec2.internal" Sep 12 01:44:56.757887 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:44:56.757861 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-79.ec2.internal\" not found" Sep 12 01:44:56.858715 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:44:56.858680 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-79.ec2.internal\" not found" Sep 12 01:44:56.871052 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.871016 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/336bcc6551e46b69eadff66cca9f19f9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-79.ec2.internal\" (UID: \"336bcc6551e46b69eadff66cca9f19f9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-79.ec2.internal" Sep 12 01:44:56.871052 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.871058 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/336bcc6551e46b69eadff66cca9f19f9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-79.ec2.internal\" (UID: \"336bcc6551e46b69eadff66cca9f19f9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-79.ec2.internal" Sep 12 01:44:56.871167 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.871076 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0f092973866d00f19dd94cc0896a8414-config\") pod \"kube-apiserver-proxy-ip-10-0-133-79.ec2.internal\" (UID: \"0f092973866d00f19dd94cc0896a8414\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-79.ec2.internal" Sep 12 01:44:56.959488 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:44:56.959416 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-79.ec2.internal\" not found" Sep 12 01:44:56.971769 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.971735 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/336bcc6551e46b69eadff66cca9f19f9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-79.ec2.internal\" (UID: \"336bcc6551e46b69eadff66cca9f19f9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-79.ec2.internal" Sep 12 01:44:56.971852 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.971776 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/336bcc6551e46b69eadff66cca9f19f9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-79.ec2.internal\" (UID: \"336bcc6551e46b69eadff66cca9f19f9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-79.ec2.internal" Sep 12 01:44:56.971852 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.971801 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0f092973866d00f19dd94cc0896a8414-config\") pod \"kube-apiserver-proxy-ip-10-0-133-79.ec2.internal\" (UID: \"0f092973866d00f19dd94cc0896a8414\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-79.ec2.internal" Sep 12 01:44:56.971852 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.971830 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/336bcc6551e46b69eadff66cca9f19f9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-79.ec2.internal\" (UID: \"336bcc6551e46b69eadff66cca9f19f9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-79.ec2.internal" Sep 12 01:44:56.971990 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.971838 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/336bcc6551e46b69eadff66cca9f19f9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-79.ec2.internal\" (UID: \"336bcc6551e46b69eadff66cca9f19f9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-79.ec2.internal" Sep 12 01:44:56.971990 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:56.971838 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0f092973866d00f19dd94cc0896a8414-config\") pod \"kube-apiserver-proxy-ip-10-0-133-79.ec2.internal\" (UID: \"0f092973866d00f19dd94cc0896a8414\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-79.ec2.internal" Sep 12 01:44:57.006811 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:57.006784 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-79.ec2.internal" Sep 12 01:44:57.011572 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:57.011389 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-79.ec2.internal" Sep 12 01:44:57.059732 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:44:57.059703 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-79.ec2.internal\" not found" Sep 12 01:44:57.160288 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:44:57.160253 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-79.ec2.internal\" not found" Sep 12 01:44:57.260816 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:44:57.260744 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-79.ec2.internal\" not found" Sep 12 01:44:57.287361 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:57.287329 2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 12 01:44:57.361599 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:44:57.361566 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-79.ec2.internal\" not found" Sep 12 01:44:57.402072 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:57.402033 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 12 01:44:57.402490 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:57.402219 2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 12 01:44:57.402490 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:57.402219 2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 12 01:44:57.462529 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:44:57.462500 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-79.ec2.internal\" not found" Sep 12 01:44:57.488513 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:57.488473 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-11 01:39:57 +0000 UTC" deadline="2028-04-28 21:51:13.564064792 +0000 UTC" Sep 12 01:44:57.488513 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:57.488505 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14276h6m16.075564666s" Sep 12 01:44:57.536359 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:57.536332 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod336bcc6551e46b69eadff66cca9f19f9.slice/crio-7631b8662006976d373dd5f0e6b47400f83dccea6aedc32ca8247c14915ee438 WatchSource:0}: Error finding container 7631b8662006976d373dd5f0e6b47400f83dccea6aedc32ca8247c14915ee438: Status 404 returned error can't find the container with id 7631b8662006976d373dd5f0e6b47400f83dccea6aedc32ca8247c14915ee438 Sep 12 01:44:57.536573 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:44:57.536558 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f092973866d00f19dd94cc0896a8414.slice/crio-6a49d0c42d903cc81b5c24726c694919bc5fccabeab82838226a343d4b378abb WatchSource:0}: Error finding container 6a49d0c42d903cc81b5c24726c694919bc5fccabeab82838226a343d4b378abb: Status 404 returned error can't find the container with id 6a49d0c42d903cc81b5c24726c694919bc5fccabeab82838226a343d4b378abb Sep 12 01:44:57.541493 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:57.541477 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 12 01:44:57.563446 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:44:57.563427 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-79.ec2.internal\" not found" Sep 12 01:44:57.571610 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:57.571580 2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 12 01:44:57.581575 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:57.581535 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-79.ec2.internal" event={"ID":"0f092973866d00f19dd94cc0896a8414","Type":"ContainerStarted","Data":"6a49d0c42d903cc81b5c24726c694919bc5fccabeab82838226a343d4b378abb"} Sep 12 01:44:57.582463 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:57.582445 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-79.ec2.internal" event={"ID":"336bcc6551e46b69eadff66cca9f19f9","Type":"ContainerStarted","Data":"7631b8662006976d373dd5f0e6b47400f83dccea6aedc32ca8247c14915ee438"} Sep 12 01:44:57.612722 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:57.612704 2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 12 01:44:57.669484 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:57.669464 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-79.ec2.internal" Sep 12 01:44:57.679215 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:57.679199 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 12 01:44:57.680313 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:57.680302 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-79.ec2.internal" Sep 12 01:44:57.689357 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:57.689340 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 12 01:44:59.588835 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.588800 2579 apiserver.go:52] "Watching apiserver" Sep 12 01:44:59.595662 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.595639 2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 12 01:44:59.598015 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.597990 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-133-79.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2shh6","openshift-dns/node-resolver-qzpw5","openshift-image-registry/node-ca-2dpmw","openshift-multus/network-metrics-daemon-nhq7s","openshift-ovn-kubernetes/ovnkube-node-mmlsx","kube-system/global-pull-secret-syncer-wjmgj","kube-system/konnectivity-agent-nhw89","openshift-cluster-node-tuning-operator/tuned-4fhmh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-79.ec2.internal","openshift-multus/multus-additional-cni-plugins-rzbs6","openshift-multus/multus-d4598","openshift-network-diagnostics/network-check-target-bxnbn","openshift-network-operator/iptables-alerter-pkjc5"] Sep 12 01:44:59.600503 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.600478 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wjmgj" Sep 12 01:44:59.600619 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:44:59.600553 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wjmgj" podUID="f016f46e-b786-405e-8f6e-faca9e639b5f" Sep 12 01:44:59.602728 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.602705 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bxnbn" Sep 12 01:44:59.602817 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:44:59.602780 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bxnbn" podUID="f02f888c-7971-4e66-8a7d-3e180b6807d9" Sep 12 01:44:59.604864 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.604841 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qzpw5" Sep 12 01:44:59.606794 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.606773 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wgttv\"" Sep 12 01:44:59.607066 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.607038 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 12 01:44:59.607066 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.607053 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 12 01:44:59.607202 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.607184 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2dpmw" Sep 12 01:44:59.609183 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.609162 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 12 01:44:59.609298 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.609242 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhq7s" Sep 12 01:44:59.609363 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:44:59.609308 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhq7s" podUID="b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03" Sep 12 01:44:59.609429 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.609384 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 12 01:44:59.609511 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.609469 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 12 01:44:59.609583 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.609556 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-r7n6p\"" Sep 12 01:44:59.610596 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.610572 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 12 01:44:59.611557 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.611541 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:44:59.613587 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.613357 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 12 01:44:59.613872 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.613855 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4fhmh" Sep 12 01:44:59.614758 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.614368 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 12 01:44:59.614758 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.614434 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 12 01:44:59.614758 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.614469 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 12 01:44:59.614758 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.614479 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 12 01:44:59.614758 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.614626 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 12 01:44:59.614758 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.614631 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bvbdj\"" Sep 12 01:44:59.615718 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.615698 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 12 01:44:59.616126 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.616100 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pkjc5" Sep 12 01:44:59.616213 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.616204 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6krwn\"" Sep 12 01:44:59.616271 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.616218 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 12 01:44:59.618156 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.618136 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 12 01:44:59.618546 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.618348 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 12 01:44:59.618546 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.618352 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 12 01:44:59.618546 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.618472 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hsmvm\"" Sep 12 01:44:59.618710 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.618700 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2shh6" Sep 12 01:44:59.622868 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.621674 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9274dec2-bf50-42c6-8c65-7ecb88771cc7-ovnkube-config\") pod \"ovnkube-node-mmlsx\" (UID: \"9274dec2-bf50-42c6-8c65-7ecb88771cc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:44:59.622868 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.621718 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9274dec2-bf50-42c6-8c65-7ecb88771cc7-run-ovn\") pod \"ovnkube-node-mmlsx\" (UID: \"9274dec2-bf50-42c6-8c65-7ecb88771cc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:44:59.622868 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.621749 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6e1eb253-a93e-4370-9cbd-a5ec2ae013a9-host\") pod \"node-ca-2dpmw\" (UID: \"6e1eb253-a93e-4370-9cbd-a5ec2ae013a9\") " pod="openshift-image-registry/node-ca-2dpmw" Sep 12 01:44:59.622868 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.621834 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcdwt\" (UniqueName: \"kubernetes.io/projected/6e1eb253-a93e-4370-9cbd-a5ec2ae013a9-kube-api-access-tcdwt\") pod \"node-ca-2dpmw\" (UID: \"6e1eb253-a93e-4370-9cbd-a5ec2ae013a9\") " pod="openshift-image-registry/node-ca-2dpmw" Sep 12 01:44:59.622868 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.621879 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 12 01:44:59.622868 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.621917 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9274dec2-bf50-42c6-8c65-7ecb88771cc7-log-socket\") pod \"ovnkube-node-mmlsx\" (UID: \"9274dec2-bf50-42c6-8c65-7ecb88771cc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:44:59.622868 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.621944 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 12 01:44:59.622868 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.621969 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-gc2gn\"" Sep 12 01:44:59.622868 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.621969 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9274dec2-bf50-42c6-8c65-7ecb88771cc7-ovnkube-script-lib\") pod \"ovnkube-node-mmlsx\" (UID: \"9274dec2-bf50-42c6-8c65-7ecb88771cc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:44:59.622868 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.622103 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f016f46e-b786-405e-8f6e-faca9e639b5f-kubelet-config\") pod \"global-pull-secret-syncer-wjmgj\" (UID: \"f016f46e-b786-405e-8f6e-faca9e639b5f\") " pod="kube-system/global-pull-secret-syncer-wjmgj" Sep 12 01:44:59.622868 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.622137 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9274dec2-bf50-42c6-8c65-7ecb88771cc7-host-kubelet\") pod \"ovnkube-node-mmlsx\" (UID: \"9274dec2-bf50-42c6-8c65-7ecb88771cc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:44:59.622868 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.622169 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9274dec2-bf50-42c6-8c65-7ecb88771cc7-systemd-units\") pod \"ovnkube-node-mmlsx\" (UID: \"9274dec2-bf50-42c6-8c65-7ecb88771cc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:44:59.622868 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.622199 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9274dec2-bf50-42c6-8c65-7ecb88771cc7-var-lib-openvswitch\") pod \"ovnkube-node-mmlsx\" (UID: \"9274dec2-bf50-42c6-8c65-7ecb88771cc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:44:59.622868 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.622230 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9274dec2-bf50-42c6-8c65-7ecb88771cc7-etc-openvswitch\") pod \"ovnkube-node-mmlsx\" (UID: \"9274dec2-bf50-42c6-8c65-7ecb88771cc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:44:59.622868 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.622255 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnlkq\" (UniqueName: \"kubernetes.io/projected/9274dec2-bf50-42c6-8c65-7ecb88771cc7-kube-api-access-jnlkq\") pod \"ovnkube-node-mmlsx\" (UID: \"9274dec2-bf50-42c6-8c65-7ecb88771cc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:44:59.622868 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.622256 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 12 01:44:59.622868 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.622302 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frwnn\" (UniqueName: \"kubernetes.io/projected/f02f888c-7971-4e66-8a7d-3e180b6807d9-kube-api-access-frwnn\") pod \"network-check-target-bxnbn\" (UID: \"f02f888c-7971-4e66-8a7d-3e180b6807d9\") " pod="openshift-network-diagnostics/network-check-target-bxnbn" Sep 12 01:44:59.622868 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.622319 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 12 01:44:59.622868 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.622337 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6e1eb253-a93e-4370-9cbd-a5ec2ae013a9-serviceca\") pod \"node-ca-2dpmw\" (UID: \"6e1eb253-a93e-4370-9cbd-a5ec2ae013a9\") " pod="openshift-image-registry/node-ca-2dpmw" Sep 12 01:44:59.623990 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.622362 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03-metrics-certs\") pod \"network-metrics-daemon-nhq7s\" (UID: \"b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03\") " pod="openshift-multus/network-metrics-daemon-nhq7s" Sep 12 01:44:59.623990 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.622387 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9274dec2-bf50-42c6-8c65-7ecb88771cc7-host-run-netns\") pod \"ovnkube-node-mmlsx\" (UID: \"9274dec2-bf50-42c6-8c65-7ecb88771cc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:44:59.623990 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.622417 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9274dec2-bf50-42c6-8c65-7ecb88771cc7-node-log\") pod \"ovnkube-node-mmlsx\" (UID: \"9274dec2-bf50-42c6-8c65-7ecb88771cc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:44:59.623990 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.622472 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9274dec2-bf50-42c6-8c65-7ecb88771cc7-env-overrides\") pod \"ovnkube-node-mmlsx\" (UID: \"9274dec2-bf50-42c6-8c65-7ecb88771cc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:44:59.623990 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.622501 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9274dec2-bf50-42c6-8c65-7ecb88771cc7-ovn-node-metrics-cert\") pod \"ovnkube-node-mmlsx\" (UID: \"9274dec2-bf50-42c6-8c65-7ecb88771cc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:44:59.623990 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.622546 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a1140d6e-61e4-4bad-a337-cc28db7dd812-hosts-file\") pod \"node-resolver-qzpw5\" (UID: \"a1140d6e-61e4-4bad-a337-cc28db7dd812\") " pod="openshift-dns/node-resolver-qzpw5" Sep 12 01:44:59.623990 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.622594 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9274dec2-bf50-42c6-8c65-7ecb88771cc7-run-systemd\") pod \"ovnkube-node-mmlsx\" (UID: \"9274dec2-bf50-42c6-8c65-7ecb88771cc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:44:59.623990 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.622626 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f016f46e-b786-405e-8f6e-faca9e639b5f-dbus\") pod \"global-pull-secret-syncer-wjmgj\" (UID: \"f016f46e-b786-405e-8f6e-faca9e639b5f\") " pod="kube-system/global-pull-secret-syncer-wjmgj" Sep 12 01:44:59.623990 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.622654 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gp9v\" (UniqueName: \"kubernetes.io/projected/b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03-kube-api-access-8gp9v\") pod \"network-metrics-daemon-nhq7s\" (UID: \"b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03\") " pod="openshift-multus/network-metrics-daemon-nhq7s" Sep 12 01:44:59.623990 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.622692 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9274dec2-bf50-42c6-8c65-7ecb88771cc7-host-slash\") pod \"ovnkube-node-mmlsx\" (UID: \"9274dec2-bf50-42c6-8c65-7ecb88771cc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:44:59.623990 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.622715 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9274dec2-bf50-42c6-8c65-7ecb88771cc7-host-run-ovn-kubernetes\") pod \"ovnkube-node-mmlsx\" (UID: \"9274dec2-bf50-42c6-8c65-7ecb88771cc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:44:59.623990 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.622742 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9274dec2-bf50-42c6-8c65-7ecb88771cc7-host-cni-netd\") pod \"ovnkube-node-mmlsx\" (UID: \"9274dec2-bf50-42c6-8c65-7ecb88771cc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:44:59.623990 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.622770 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f016f46e-b786-405e-8f6e-faca9e639b5f-original-pull-secret\") pod \"global-pull-secret-syncer-wjmgj\" (UID: \"f016f46e-b786-405e-8f6e-faca9e639b5f\") " pod="kube-system/global-pull-secret-syncer-wjmgj" Sep 12 01:44:59.623990 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.622811 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1140d6e-61e4-4bad-a337-cc28db7dd812-tmp-dir\") pod \"node-resolver-qzpw5\" (UID: \"a1140d6e-61e4-4bad-a337-cc28db7dd812\") " pod="openshift-dns/node-resolver-qzpw5" Sep 12 01:44:59.623990 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.622834 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkn6j\" (UniqueName: \"kubernetes.io/projected/a1140d6e-61e4-4bad-a337-cc28db7dd812-kube-api-access-nkn6j\") pod \"node-resolver-qzpw5\" (UID: \"a1140d6e-61e4-4bad-a337-cc28db7dd812\") " pod="openshift-dns/node-resolver-qzpw5" Sep 12 01:44:59.623990 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.622953 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9274dec2-bf50-42c6-8c65-7ecb88771cc7-run-openvswitch\") pod \"ovnkube-node-mmlsx\" (UID: \"9274dec2-bf50-42c6-8c65-7ecb88771cc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:44:59.624549 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.622986 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9274dec2-bf50-42c6-8c65-7ecb88771cc7-host-cni-bin\") pod \"ovnkube-node-mmlsx\" (UID: \"9274dec2-bf50-42c6-8c65-7ecb88771cc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:44:59.624549 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.623055 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9274dec2-bf50-42c6-8c65-7ecb88771cc7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mmlsx\" (UID: \"9274dec2-bf50-42c6-8c65-7ecb88771cc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:44:59.624549 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.624234 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nhw89" Sep 12 01:44:59.626322 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.626305 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 12 01:44:59.626578 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.626562 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 12 01:44:59.626696 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.626625 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-cbpvq\"" Sep 12 01:44:59.627156 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.626917 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rzbs6" Sep 12 01:44:59.627940 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.627921 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 12 01:44:59.629372 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.629118 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-d4598" Sep 12 01:44:59.629372 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.629276 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 12 01:44:59.629372 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.629290 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 12 01:44:59.629600 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.629585 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 12 01:44:59.629831 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.629584 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 12 01:44:59.629831 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.629742 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nr4ld\"" Sep 12 01:44:59.629831 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.629818 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 12 01:44:59.631085 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.631067 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 12 01:44:59.631085 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.631081 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-bb27x\"" Sep 12 01:44:59.643337 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.643319 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-d7ksx" Sep 12 01:44:59.651609 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.651577 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-d7ksx" Sep 12 01:44:59.712137 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.712110 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 12 01:44:59.723718 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.723692 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9274dec2-bf50-42c6-8c65-7ecb88771cc7-log-socket\") pod \"ovnkube-node-mmlsx\" (UID: \"9274dec2-bf50-42c6-8c65-7ecb88771cc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:44:59.723845 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.723726 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9274dec2-bf50-42c6-8c65-7ecb88771cc7-ovnkube-script-lib\") pod \"ovnkube-node-mmlsx\" (UID: \"9274dec2-bf50-42c6-8c65-7ecb88771cc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:44:59.723845 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.723788 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9274dec2-bf50-42c6-8c65-7ecb88771cc7-log-socket\") pod \"ovnkube-node-mmlsx\" (UID: \"9274dec2-bf50-42c6-8c65-7ecb88771cc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:44:59.723958 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.723841 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f016f46e-b786-405e-8f6e-faca9e639b5f-kubelet-config\") pod \"global-pull-secret-syncer-wjmgj\" (UID: \"f016f46e-b786-405e-8f6e-faca9e639b5f\") " pod="kube-system/global-pull-secret-syncer-wjmgj" Sep 12 01:44:59.723958 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.723879 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5a68d964-2017-4185-b57a-5b09bfe19c4f-etc-tuned\") pod \"tuned-4fhmh\" (UID: \"5a68d964-2017-4185-b57a-5b09bfe19c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4fhmh" Sep 12 01:44:59.723958 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.723911 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/900016e2-2d16-467b-a924-83f1aec3610e-host-slash\") pod \"iptables-alerter-pkjc5\" (UID: \"900016e2-2d16-467b-a924-83f1aec3610e\") " pod="openshift-network-operator/iptables-alerter-pkjc5" Sep 12 01:44:59.723958 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.723935 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5a68d964-2017-4185-b57a-5b09bfe19c4f-etc-sysctl-conf\") pod \"tuned-4fhmh\" (UID: \"5a68d964-2017-4185-b57a-5b09bfe19c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4fhmh" Sep 12 01:44:59.723958 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.723935 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f016f46e-b786-405e-8f6e-faca9e639b5f-kubelet-config\") pod \"global-pull-secret-syncer-wjmgj\" (UID: \"f016f46e-b786-405e-8f6e-faca9e639b5f\") " pod="kube-system/global-pull-secret-syncer-wjmgj" Sep 12 01:44:59.724147 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.723961 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9274dec2-bf50-42c6-8c65-7ecb88771cc7-host-kubelet\") pod \"ovnkube-node-mmlsx\" (UID: \"9274dec2-bf50-42c6-8c65-7ecb88771cc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:44:59.724147 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.723984 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9274dec2-bf50-42c6-8c65-7ecb88771cc7-systemd-units\") pod \"ovnkube-node-mmlsx\" (UID: \"9274dec2-bf50-42c6-8c65-7ecb88771cc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:44:59.724147 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.724006 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9274dec2-bf50-42c6-8c65-7ecb88771cc7-var-lib-openvswitch\") pod \"ovnkube-node-mmlsx\" (UID: \"9274dec2-bf50-42c6-8c65-7ecb88771cc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:44:59.724147 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.724030 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9274dec2-bf50-42c6-8c65-7ecb88771cc7-etc-openvswitch\") pod \"ovnkube-node-mmlsx\" (UID: \"9274dec2-bf50-42c6-8c65-7ecb88771cc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:44:59.724147 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.724056 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnlkq\" (UniqueName: \"kubernetes.io/projected/9274dec2-bf50-42c6-8c65-7ecb88771cc7-kube-api-access-jnlkq\") pod \"ovnkube-node-mmlsx\" (UID: \"9274dec2-bf50-42c6-8c65-7ecb88771cc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:44:59.724147 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.724075 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9274dec2-bf50-42c6-8c65-7ecb88771cc7-systemd-units\") pod \"ovnkube-node-mmlsx\" (UID: \"9274dec2-bf50-42c6-8c65-7ecb88771cc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:44:59.724147 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.724105 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frwnn\" (UniqueName: \"kubernetes.io/projected/f02f888c-7971-4e66-8a7d-3e180b6807d9-kube-api-access-frwnn\") pod \"network-check-target-bxnbn\" (UID: \"f02f888c-7971-4e66-8a7d-3e180b6807d9\") " pod="openshift-network-diagnostics/network-check-target-bxnbn" Sep 12 01:44:59.724147 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.724129 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9274dec2-bf50-42c6-8c65-7ecb88771cc7-var-lib-openvswitch\") pod \"ovnkube-node-mmlsx\" (UID: \"9274dec2-bf50-42c6-8c65-7ecb88771cc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:44:59.724395 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.724137 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9274dec2-bf50-42c6-8c65-7ecb88771cc7-host-kubelet\") pod \"ovnkube-node-mmlsx\" (UID: \"9274dec2-bf50-42c6-8c65-7ecb88771cc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:44:59.724395 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.724155 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6e1eb253-a93e-4370-9cbd-a5ec2ae013a9-serviceca\") pod \"node-ca-2dpmw\" (UID: \"6e1eb253-a93e-4370-9cbd-a5ec2ae013a9\") " pod="openshift-image-registry/node-ca-2dpmw" Sep 12 01:44:59.724395 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.724181 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03-metrics-certs\") pod \"network-metrics-daemon-nhq7s\" (UID: \"b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03\") " pod="openshift-multus/network-metrics-daemon-nhq7s" Sep 12 01:44:59.724395 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.724200 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9274dec2-bf50-42c6-8c65-7ecb88771cc7-etc-openvswitch\") pod \"ovnkube-node-mmlsx\" (UID: \"9274dec2-bf50-42c6-8c65-7ecb88771cc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:44:59.724395 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.724206 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9274dec2-bf50-42c6-8c65-7ecb88771cc7-host-run-netns\") pod \"ovnkube-node-mmlsx\" (UID: \"9274dec2-bf50-42c6-8c65-7ecb88771cc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:44:59.724395 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.724251 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9274dec2-bf50-42c6-8c65-7ecb88771cc7-env-overrides\") pod \"ovnkube-node-mmlsx\" (UID: \"9274dec2-bf50-42c6-8c65-7ecb88771cc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:44:59.724395 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.724272 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9274dec2-bf50-42c6-8c65-7ecb88771cc7-ovn-node-metrics-cert\") pod \"ovnkube-node-mmlsx\" (UID: \"9274dec2-bf50-42c6-8c65-7ecb88771cc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:44:59.724395 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.724297 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e5ed1052-5fa4-4d50-be77-d799d5722140-host-run-k8s-cni-cncf-io\") pod \"multus-d4598\" (UID: \"e5ed1052-5fa4-4d50-be77-d799d5722140\") " pod="openshift-multus/multus-d4598" Sep 12 01:44:59.724395 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.724325 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5a68d964-2017-4185-b57a-5b09bfe19c4f-lib-modules\") pod \"tuned-4fhmh\" (UID: \"5a68d964-2017-4185-b57a-5b09bfe19c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4fhmh" Sep 12 01:44:59.724395 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.724354 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9274dec2-bf50-42c6-8c65-7ecb88771cc7-ovnkube-script-lib\") pod \"ovnkube-node-mmlsx\" (UID: \"9274dec2-bf50-42c6-8c65-7ecb88771cc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:44:59.724395 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:44:59.724365 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 12 01:44:59.724775 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:44:59.724416 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03-metrics-certs podName:b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03 nodeName:}" failed. No retries permitted until 2026-09-12 01:45:00.22439823 +0000 UTC m=+3.016392036 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03-metrics-certs") pod "network-metrics-daemon-nhq7s" (UID: "b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 12 01:44:59.724775 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.724418 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9274dec2-bf50-42c6-8c65-7ecb88771cc7-host-run-netns\") pod \"ovnkube-node-mmlsx\" (UID: \"9274dec2-bf50-42c6-8c65-7ecb88771cc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:44:59.724775 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.724364 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-994lz\" (UniqueName: \"kubernetes.io/projected/5a68d964-2017-4185-b57a-5b09bfe19c4f-kube-api-access-994lz\") pod \"tuned-4fhmh\" (UID: \"5a68d964-2017-4185-b57a-5b09bfe19c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4fhmh" Sep 12 01:44:59.724775 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.724502 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e7d6653c-acc4-4af4-89c4-39eff31ba128-socket-dir\") pod \"aws-ebs-csi-driver-node-2shh6\" (UID: \"e7d6653c-acc4-4af4-89c4-39eff31ba128\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2shh6" Sep 12 01:44:59.724775 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.724530 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9274dec2-bf50-42c6-8c65-7ecb88771cc7-run-systemd\") pod \"ovnkube-node-mmlsx\" (UID: \"9274dec2-bf50-42c6-8c65-7ecb88771cc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:44:59.724775 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.724555 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f016f46e-b786-405e-8f6e-faca9e639b5f-dbus\") pod \"global-pull-secret-syncer-wjmgj\" (UID: \"f016f46e-b786-405e-8f6e-faca9e639b5f\") " pod="kube-system/global-pull-secret-syncer-wjmgj" Sep 12 01:44:59.724775 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.724584 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gp9v\" (UniqueName: \"kubernetes.io/projected/b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03-kube-api-access-8gp9v\") pod \"network-metrics-daemon-nhq7s\" (UID: \"b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03\") " pod="openshift-multus/network-metrics-daemon-nhq7s" Sep 12 01:44:59.724775 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.724584 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9274dec2-bf50-42c6-8c65-7ecb88771cc7-run-systemd\") pod \"ovnkube-node-mmlsx\" (UID: \"9274dec2-bf50-42c6-8c65-7ecb88771cc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:44:59.724775 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.724652 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6e1eb253-a93e-4370-9cbd-a5ec2ae013a9-serviceca\") pod \"node-ca-2dpmw\" (UID: \"6e1eb253-a93e-4370-9cbd-a5ec2ae013a9\") " pod="openshift-image-registry/node-ca-2dpmw" Sep 12 01:44:59.724775 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.724607 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5a68d964-2017-4185-b57a-5b09bfe19c4f-etc-systemd\") pod \"tuned-4fhmh\" (UID: \"5a68d964-2017-4185-b57a-5b09bfe19c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4fhmh" Sep 12 01:44:59.724775 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.724694 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a68d964-2017-4185-b57a-5b09bfe19c4f-host\") pod \"tuned-4fhmh\" (UID: \"5a68d964-2017-4185-b57a-5b09bfe19c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4fhmh" Sep 12 01:44:59.724775 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.724727 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e7d6653c-acc4-4af4-89c4-39eff31ba128-device-dir\") pod \"aws-ebs-csi-driver-node-2shh6\" (UID: \"e7d6653c-acc4-4af4-89c4-39eff31ba128\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2shh6" Sep 12 01:44:59.724775 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.724751 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e7d6653c-acc4-4af4-89c4-39eff31ba128-sys-fs\") pod \"aws-ebs-csi-driver-node-2shh6\" (UID: \"e7d6653c-acc4-4af4-89c4-39eff31ba128\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2shh6" Sep 12 01:44:59.724775 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.724756 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f016f46e-b786-405e-8f6e-faca9e639b5f-dbus\") pod \"global-pull-secret-syncer-wjmgj\" (UID: \"f016f46e-b786-405e-8f6e-faca9e639b5f\") " pod="kube-system/global-pull-secret-syncer-wjmgj" Sep 12 01:44:59.725208 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.724783 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e5ed1052-5fa4-4d50-be77-d799d5722140-multus-daemon-config\") pod \"multus-d4598\" (UID: \"e5ed1052-5fa4-4d50-be77-d799d5722140\") " pod="openshift-multus/multus-d4598" Sep 12 01:44:59.725208 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.724808 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f016f46e-b786-405e-8f6e-faca9e639b5f-original-pull-secret\") pod \"global-pull-secret-syncer-wjmgj\" (UID: \"f016f46e-b786-405e-8f6e-faca9e639b5f\") " pod="kube-system/global-pull-secret-syncer-wjmgj" Sep 12 01:44:59.725208 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.724822 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e5ed1052-5fa4-4d50-be77-d799d5722140-multus-socket-dir-parent\") pod \"multus-d4598\" (UID: \"e5ed1052-5fa4-4d50-be77-d799d5722140\") " pod="openshift-multus/multus-d4598" Sep 12 01:44:59.725208 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.724839 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/46256f89-0730-441e-97d5-6ef1dff74569-system-cni-dir\") pod \"multus-additional-cni-plugins-rzbs6\" (UID: \"46256f89-0730-441e-97d5-6ef1dff74569\") " pod="openshift-multus/multus-additional-cni-plugins-rzbs6" Sep 12 01:44:59.725208 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.724862 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/46256f89-0730-441e-97d5-6ef1dff74569-cni-binary-copy\") pod \"multus-additional-cni-plugins-rzbs6\" (UID: \"46256f89-0730-441e-97d5-6ef1dff74569\") " pod="openshift-multus/multus-additional-cni-plugins-rzbs6" Sep 12 01:44:59.725208 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.725002 2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 12 01:44:59.725208 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:44:59.725069 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 12 01:44:59.725208 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:44:59.725145 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f016f46e-b786-405e-8f6e-faca9e639b5f-original-pull-secret podName:f016f46e-b786-405e-8f6e-faca9e639b5f nodeName:}" failed. No retries permitted until 2026-09-12 01:45:00.225126432 +0000 UTC m=+3.017120231 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f016f46e-b786-405e-8f6e-faca9e639b5f-original-pull-secret") pod "global-pull-secret-syncer-wjmgj" (UID: "f016f46e-b786-405e-8f6e-faca9e639b5f") : object "kube-system"/"original-pull-secret" not registered Sep 12 01:44:59.725208 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.725175 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9274dec2-bf50-42c6-8c65-7ecb88771cc7-env-overrides\") pod \"ovnkube-node-mmlsx\" (UID: \"9274dec2-bf50-42c6-8c65-7ecb88771cc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:44:59.725662 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.725331 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/46256f89-0730-441e-97d5-6ef1dff74569-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rzbs6\" (UID: \"46256f89-0730-441e-97d5-6ef1dff74569\") " pod="openshift-multus/multus-additional-cni-plugins-rzbs6" Sep 12 01:44:59.725662 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.725353 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbs9d\" (UniqueName: \"kubernetes.io/projected/46256f89-0730-441e-97d5-6ef1dff74569-kube-api-access-kbs9d\") pod \"multus-additional-cni-plugins-rzbs6\" (UID: \"46256f89-0730-441e-97d5-6ef1dff74569\") " pod="openshift-multus/multus-additional-cni-plugins-rzbs6" Sep 12 01:44:59.725662 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.725372 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5a68d964-2017-4185-b57a-5b09bfe19c4f-run\") pod \"tuned-4fhmh\" (UID: \"5a68d964-2017-4185-b57a-5b09bfe19c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4fhmh" Sep 12 01:44:59.725662 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.725387 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr2wt\" (UniqueName: \"kubernetes.io/projected/e7d6653c-acc4-4af4-89c4-39eff31ba128-kube-api-access-fr2wt\") pod \"aws-ebs-csi-driver-node-2shh6\" (UID: \"e7d6653c-acc4-4af4-89c4-39eff31ba128\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2shh6" Sep 12 01:44:59.725662 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.725402 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5ed1052-5fa4-4d50-be77-d799d5722140-host-var-lib-kubelet\") pod \"multus-d4598\" (UID: \"e5ed1052-5fa4-4d50-be77-d799d5722140\") " pod="openshift-multus/multus-d4598" Sep 12 01:44:59.725662 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.725420 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5a68d964-2017-4185-b57a-5b09bfe19c4f-etc-sysconfig\") pod \"tuned-4fhmh\" (UID: \"5a68d964-2017-4185-b57a-5b09bfe19c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4fhmh" Sep 12 01:44:59.725662 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.725437 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a68d964-2017-4185-b57a-5b09bfe19c4f-etc-kubernetes\") pod \"tuned-4fhmh\" (UID: \"5a68d964-2017-4185-b57a-5b09bfe19c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4fhmh" Sep 12 01:44:59.725662 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.725475 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5a68d964-2017-4185-b57a-5b09bfe19c4f-etc-sysctl-d\") pod \"tuned-4fhmh\" (UID: \"5a68d964-2017-4185-b57a-5b09bfe19c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4fhmh" Sep 12 01:44:59.725662 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.725497 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/900016e2-2d16-467b-a924-83f1aec3610e-iptables-alerter-script\") pod \"iptables-alerter-pkjc5\" (UID: \"900016e2-2d16-467b-a924-83f1aec3610e\") " pod="openshift-network-operator/iptables-alerter-pkjc5" Sep 12 01:44:59.725662 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.725519 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e7d6653c-acc4-4af4-89c4-39eff31ba128-etc-selinux\") pod \"aws-ebs-csi-driver-node-2shh6\" (UID: \"e7d6653c-acc4-4af4-89c4-39eff31ba128\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2shh6" Sep 12 01:44:59.725662 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.725547 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6e1eb253-a93e-4370-9cbd-a5ec2ae013a9-host\") pod \"node-ca-2dpmw\" (UID: \"6e1eb253-a93e-4370-9cbd-a5ec2ae013a9\") " pod="openshift-image-registry/node-ca-2dpmw" Sep 12 01:44:59.725662 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.725583 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/46256f89-0730-441e-97d5-6ef1dff74569-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rzbs6\" (UID: \"46256f89-0730-441e-97d5-6ef1dff74569\") " pod="openshift-multus/multus-additional-cni-plugins-rzbs6" Sep 12 01:44:59.725662 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.725606 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5ed1052-5fa4-4d50-be77-d799d5722140-multus-cni-dir\") pod \"multus-d4598\" (UID: \"e5ed1052-5fa4-4d50-be77-d799d5722140\") " pod="openshift-multus/multus-d4598" Sep 12 01:44:59.725662 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.725626 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e5ed1052-5fa4-4d50-be77-d799d5722140-cni-binary-copy\") pod \"multus-d4598\" (UID: \"e5ed1052-5fa4-4d50-be77-d799d5722140\") " pod="openshift-multus/multus-d4598" Sep 12 01:44:59.725662 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.725647 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmcgq\" (UniqueName: \"kubernetes.io/projected/e5ed1052-5fa4-4d50-be77-d799d5722140-kube-api-access-gmcgq\") pod \"multus-d4598\" (UID: \"e5ed1052-5fa4-4d50-be77-d799d5722140\") " pod="openshift-multus/multus-d4598" Sep 12 01:44:59.726234 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.725670 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5a68d964-2017-4185-b57a-5b09bfe19c4f-etc-modprobe-d\") pod \"tuned-4fhmh\" (UID: \"5a68d964-2017-4185-b57a-5b09bfe19c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4fhmh" Sep 12 01:44:59.726234 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.725691 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e5ed1052-5fa4-4d50-be77-d799d5722140-cnibin\") pod \"multus-d4598\" (UID: \"e5ed1052-5fa4-4d50-be77-d799d5722140\") " pod="openshift-multus/multus-d4598" Sep 12 01:44:59.726234 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.725716 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e7d6653c-acc4-4af4-89c4-39eff31ba128-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2shh6\" (UID: \"e7d6653c-acc4-4af4-89c4-39eff31ba128\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2shh6" Sep 12 01:44:59.726234 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.725743 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9274dec2-bf50-42c6-8c65-7ecb88771cc7-node-log\") pod \"ovnkube-node-mmlsx\" (UID: \"9274dec2-bf50-42c6-8c65-7ecb88771cc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:44:59.726234 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.725753 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6e1eb253-a93e-4370-9cbd-a5ec2ae013a9-host\") pod \"node-ca-2dpmw\" (UID: \"6e1eb253-a93e-4370-9cbd-a5ec2ae013a9\") " pod="openshift-image-registry/node-ca-2dpmw" Sep 12 01:44:59.726234 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.725781 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a1140d6e-61e4-4bad-a337-cc28db7dd812-hosts-file\") pod \"node-resolver-qzpw5\" (UID: \"a1140d6e-61e4-4bad-a337-cc28db7dd812\") " pod="openshift-dns/node-resolver-qzpw5" Sep 12 01:44:59.726234 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.725799 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9274dec2-bf50-42c6-8c65-7ecb88771cc7-node-log\") pod \"ovnkube-node-mmlsx\" (UID: \"9274dec2-bf50-42c6-8c65-7ecb88771cc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:44:59.726234 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.725831 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e5ed1052-5fa4-4d50-be77-d799d5722140-host-var-lib-cni-bin\") pod \"multus-d4598\" (UID: \"e5ed1052-5fa4-4d50-be77-d799d5722140\") " pod="openshift-multus/multus-d4598" Sep 12 01:44:59.726234 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.725853 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e5ed1052-5fa4-4d50-be77-d799d5722140-hostroot\") pod \"multus-d4598\" (UID: \"e5ed1052-5fa4-4d50-be77-d799d5722140\") " pod="openshift-multus/multus-d4598" Sep 12 01:44:59.726234 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.725889 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a68d964-2017-4185-b57a-5b09bfe19c4f-var-lib-kubelet\") pod \"tuned-4fhmh\" (UID: \"5a68d964-2017-4185-b57a-5b09bfe19c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4fhmh" Sep 12 01:44:59.726234 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.725906 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a1140d6e-61e4-4bad-a337-cc28db7dd812-hosts-file\") pod \"node-resolver-qzpw5\" (UID: \"a1140d6e-61e4-4bad-a337-cc28db7dd812\") " pod="openshift-dns/node-resolver-qzpw5" Sep 12 01:44:59.726234 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.725924 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5ed1052-5fa4-4d50-be77-d799d5722140-etc-kubernetes\") pod \"multus-d4598\" (UID: \"e5ed1052-5fa4-4d50-be77-d799d5722140\") " pod="openshift-multus/multus-d4598" Sep 12 01:44:59.726234 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.725966 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a68d964-2017-4185-b57a-5b09bfe19c4f-tmp\") pod \"tuned-4fhmh\" (UID: \"5a68d964-2017-4185-b57a-5b09bfe19c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4fhmh" Sep 12 01:44:59.726234 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.725991 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbzft\" (UniqueName: \"kubernetes.io/projected/900016e2-2d16-467b-a924-83f1aec3610e-kube-api-access-qbzft\") pod \"iptables-alerter-pkjc5\" (UID: \"900016e2-2d16-467b-a924-83f1aec3610e\") " pod="openshift-network-operator/iptables-alerter-pkjc5" Sep 12 01:44:59.726234 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.726018 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9274dec2-bf50-42c6-8c65-7ecb88771cc7-host-slash\") pod \"ovnkube-node-mmlsx\" (UID: \"9274dec2-bf50-42c6-8c65-7ecb88771cc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:44:59.726234 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.726048 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9274dec2-bf50-42c6-8c65-7ecb88771cc7-host-run-ovn-kubernetes\") pod \"ovnkube-node-mmlsx\" (UID: \"9274dec2-bf50-42c6-8c65-7ecb88771cc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:44:59.726234 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.726085 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9274dec2-bf50-42c6-8c65-7ecb88771cc7-host-slash\") pod \"ovnkube-node-mmlsx\" (UID: \"9274dec2-bf50-42c6-8c65-7ecb88771cc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:44:59.727012 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.726128 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9274dec2-bf50-42c6-8c65-7ecb88771cc7-host-cni-netd\") pod \"ovnkube-node-mmlsx\" (UID: \"9274dec2-bf50-42c6-8c65-7ecb88771cc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:44:59.727012 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.726154 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1140d6e-61e4-4bad-a337-cc28db7dd812-tmp-dir\") pod \"node-resolver-qzpw5\" (UID: \"a1140d6e-61e4-4bad-a337-cc28db7dd812\") " pod="openshift-dns/node-resolver-qzpw5" Sep 12 01:44:59.727012 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.726158 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9274dec2-bf50-42c6-8c65-7ecb88771cc7-host-cni-netd\") pod \"ovnkube-node-mmlsx\" (UID: \"9274dec2-bf50-42c6-8c65-7ecb88771cc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:44:59.727012 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.726179 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkn6j\" (UniqueName: \"kubernetes.io/projected/a1140d6e-61e4-4bad-a337-cc28db7dd812-kube-api-access-nkn6j\") pod \"node-resolver-qzpw5\" (UID: \"a1140d6e-61e4-4bad-a337-cc28db7dd812\") " pod="openshift-dns/node-resolver-qzpw5" Sep 12 01:44:59.727012 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.726181 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9274dec2-bf50-42c6-8c65-7ecb88771cc7-host-run-ovn-kubernetes\") pod \"ovnkube-node-mmlsx\" (UID: \"9274dec2-bf50-42c6-8c65-7ecb88771cc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:44:59.727012 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.726288 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e5ed1052-5fa4-4d50-be77-d799d5722140-multus-conf-dir\") pod \"multus-d4598\" (UID: \"e5ed1052-5fa4-4d50-be77-d799d5722140\") " pod="openshift-multus/multus-d4598" Sep 12 01:44:59.727012 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.726373 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9274dec2-bf50-42c6-8c65-7ecb88771cc7-run-openvswitch\") pod \"ovnkube-node-mmlsx\" (UID: \"9274dec2-bf50-42c6-8c65-7ecb88771cc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:44:59.727012 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.726401 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9274dec2-bf50-42c6-8c65-7ecb88771cc7-host-cni-bin\") pod \"ovnkube-node-mmlsx\" (UID: \"9274dec2-bf50-42c6-8c65-7ecb88771cc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:44:59.727012 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.726424 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9274dec2-bf50-42c6-8c65-7ecb88771cc7-run-openvswitch\") pod \"ovnkube-node-mmlsx\" (UID: \"9274dec2-bf50-42c6-8c65-7ecb88771cc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:44:59.727012 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.726432 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1140d6e-61e4-4bad-a337-cc28db7dd812-tmp-dir\") pod \"node-resolver-qzpw5\" (UID: \"a1140d6e-61e4-4bad-a337-cc28db7dd812\") " pod="openshift-dns/node-resolver-qzpw5" Sep 12 01:44:59.727012 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.726492 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9274dec2-bf50-42c6-8c65-7ecb88771cc7-host-cni-bin\") pod \"ovnkube-node-mmlsx\" (UID: \"9274dec2-bf50-42c6-8c65-7ecb88771cc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:44:59.727012 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.726503 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9274dec2-bf50-42c6-8c65-7ecb88771cc7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mmlsx\" (UID: \"9274dec2-bf50-42c6-8c65-7ecb88771cc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:44:59.727012 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.726530 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e5ed1052-5fa4-4d50-be77-d799d5722140-host-run-multus-certs\") pod \"multus-d4598\" (UID: \"e5ed1052-5fa4-4d50-be77-d799d5722140\") " pod="openshift-multus/multus-d4598" Sep 12 01:44:59.727012 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.726550 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b1408eab-ddcd-41d6-87af-50980dd43721-agent-certs\") pod \"konnectivity-agent-nhw89\" (UID: \"b1408eab-ddcd-41d6-87af-50980dd43721\") " pod="kube-system/konnectivity-agent-nhw89" Sep 12 01:44:59.727012 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.726598 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b1408eab-ddcd-41d6-87af-50980dd43721-konnectivity-ca\") pod \"konnectivity-agent-nhw89\" (UID: \"b1408eab-ddcd-41d6-87af-50980dd43721\") " pod="kube-system/konnectivity-agent-nhw89" Sep 12 01:44:59.727012 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.726604 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9274dec2-bf50-42c6-8c65-7ecb88771cc7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mmlsx\" (UID: \"9274dec2-bf50-42c6-8c65-7ecb88771cc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:44:59.727012 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.726627 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9274dec2-bf50-42c6-8c65-7ecb88771cc7-ovnkube-config\") pod \"ovnkube-node-mmlsx\" (UID: \"9274dec2-bf50-42c6-8c65-7ecb88771cc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:44:59.727677 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.726660 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5ed1052-5fa4-4d50-be77-d799d5722140-system-cni-dir\") pod \"multus-d4598\" (UID: \"e5ed1052-5fa4-4d50-be77-d799d5722140\") " pod="openshift-multus/multus-d4598" Sep 12 01:44:59.727677 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.726686 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e5ed1052-5fa4-4d50-be77-d799d5722140-host-var-lib-cni-multus\") pod \"multus-d4598\" (UID: \"e5ed1052-5fa4-4d50-be77-d799d5722140\") " pod="openshift-multus/multus-d4598" Sep 12 01:44:59.727677 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.726710 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/46256f89-0730-441e-97d5-6ef1dff74569-cnibin\") pod \"multus-additional-cni-plugins-rzbs6\" (UID: \"46256f89-0730-441e-97d5-6ef1dff74569\") " pod="openshift-multus/multus-additional-cni-plugins-rzbs6" Sep 12 01:44:59.727677 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.726736 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9274dec2-bf50-42c6-8c65-7ecb88771cc7-run-ovn\") pod \"ovnkube-node-mmlsx\" (UID: \"9274dec2-bf50-42c6-8c65-7ecb88771cc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:44:59.727677 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.726761 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcdwt\" (UniqueName: \"kubernetes.io/projected/6e1eb253-a93e-4370-9cbd-a5ec2ae013a9-kube-api-access-tcdwt\") pod \"node-ca-2dpmw\" (UID: \"6e1eb253-a93e-4370-9cbd-a5ec2ae013a9\") " pod="openshift-image-registry/node-ca-2dpmw" Sep 12 01:44:59.727677 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.726784 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e5ed1052-5fa4-4d50-be77-d799d5722140-host-run-netns\") pod \"multus-d4598\" (UID: \"e5ed1052-5fa4-4d50-be77-d799d5722140\") " pod="openshift-multus/multus-d4598" Sep 12 01:44:59.727677 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.726808 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a68d964-2017-4185-b57a-5b09bfe19c4f-sys\") pod \"tuned-4fhmh\" (UID: \"5a68d964-2017-4185-b57a-5b09bfe19c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4fhmh" Sep 12 01:44:59.727677 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.726832 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e7d6653c-acc4-4af4-89c4-39eff31ba128-registration-dir\") pod \"aws-ebs-csi-driver-node-2shh6\" (UID: \"e7d6653c-acc4-4af4-89c4-39eff31ba128\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2shh6" Sep 12 01:44:59.727677 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.726857 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/46256f89-0730-441e-97d5-6ef1dff74569-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rzbs6\" (UID: \"46256f89-0730-441e-97d5-6ef1dff74569\") " pod="openshift-multus/multus-additional-cni-plugins-rzbs6" Sep 12 01:44:59.727677 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.726949 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9274dec2-bf50-42c6-8c65-7ecb88771cc7-run-ovn\") pod \"ovnkube-node-mmlsx\" (UID: \"9274dec2-bf50-42c6-8c65-7ecb88771cc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:44:59.727677 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.726990 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9274dec2-bf50-42c6-8c65-7ecb88771cc7-ovnkube-config\") pod \"ovnkube-node-mmlsx\" (UID: \"9274dec2-bf50-42c6-8c65-7ecb88771cc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:44:59.731422 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:44:59.730589 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 12 01:44:59.731422 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:44:59.730612 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 12 01:44:59.731422 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:44:59.730625 2579 projected.go:196] Error preparing data for projected volume kube-api-access-frwnn for pod openshift-network-diagnostics/network-check-target-bxnbn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 12 01:44:59.731422 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:44:59.730698 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f02f888c-7971-4e66-8a7d-3e180b6807d9-kube-api-access-frwnn podName:f02f888c-7971-4e66-8a7d-3e180b6807d9 nodeName:}" failed. No retries permitted until 2026-09-12 01:45:00.230682409 +0000 UTC m=+3.022676199 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-frwnn" (UniqueName: "kubernetes.io/projected/f02f888c-7971-4e66-8a7d-3e180b6807d9-kube-api-access-frwnn") pod "network-check-target-bxnbn" (UID: "f02f888c-7971-4e66-8a7d-3e180b6807d9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 12 01:44:59.731422 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.730688 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9274dec2-bf50-42c6-8c65-7ecb88771cc7-ovn-node-metrics-cert\") pod \"ovnkube-node-mmlsx\" (UID: \"9274dec2-bf50-42c6-8c65-7ecb88771cc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:44:59.734673 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.734649 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnlkq\" (UniqueName: \"kubernetes.io/projected/9274dec2-bf50-42c6-8c65-7ecb88771cc7-kube-api-access-jnlkq\") pod \"ovnkube-node-mmlsx\" (UID: \"9274dec2-bf50-42c6-8c65-7ecb88771cc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:44:59.734788 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.734683 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkn6j\" (UniqueName: \"kubernetes.io/projected/a1140d6e-61e4-4bad-a337-cc28db7dd812-kube-api-access-nkn6j\") pod \"node-resolver-qzpw5\" (UID: \"a1140d6e-61e4-4bad-a337-cc28db7dd812\") " pod="openshift-dns/node-resolver-qzpw5" Sep 12 01:44:59.735558 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.735539 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcdwt\" (UniqueName: \"kubernetes.io/projected/6e1eb253-a93e-4370-9cbd-a5ec2ae013a9-kube-api-access-tcdwt\") pod \"node-ca-2dpmw\" (UID: \"6e1eb253-a93e-4370-9cbd-a5ec2ae013a9\") " pod="openshift-image-registry/node-ca-2dpmw" Sep 12 01:44:59.738011 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.737992 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gp9v\" (UniqueName: \"kubernetes.io/projected/b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03-kube-api-access-8gp9v\") pod \"network-metrics-daemon-nhq7s\" (UID: \"b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03\") " pod="openshift-multus/network-metrics-daemon-nhq7s" Sep 12 01:44:59.827083 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.827056 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e7d6653c-acc4-4af4-89c4-39eff31ba128-device-dir\") pod \"aws-ebs-csi-driver-node-2shh6\" (UID: \"e7d6653c-acc4-4af4-89c4-39eff31ba128\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2shh6" Sep 12 01:44:59.827083 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.827083 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e7d6653c-acc4-4af4-89c4-39eff31ba128-sys-fs\") pod \"aws-ebs-csi-driver-node-2shh6\" (UID: \"e7d6653c-acc4-4af4-89c4-39eff31ba128\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2shh6" Sep 12 01:44:59.827309 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.827099 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e5ed1052-5fa4-4d50-be77-d799d5722140-multus-daemon-config\") pod \"multus-d4598\" (UID: \"e5ed1052-5fa4-4d50-be77-d799d5722140\") " pod="openshift-multus/multus-d4598" Sep 12 01:44:59.827309 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.827159 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e7d6653c-acc4-4af4-89c4-39eff31ba128-sys-fs\") pod \"aws-ebs-csi-driver-node-2shh6\" (UID: \"e7d6653c-acc4-4af4-89c4-39eff31ba128\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2shh6" Sep 12 01:44:59.827309 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.827168 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e7d6653c-acc4-4af4-89c4-39eff31ba128-device-dir\") pod \"aws-ebs-csi-driver-node-2shh6\" (UID: \"e7d6653c-acc4-4af4-89c4-39eff31ba128\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2shh6" Sep 12 01:44:59.827309 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.827183 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e5ed1052-5fa4-4d50-be77-d799d5722140-multus-socket-dir-parent\") pod \"multus-d4598\" (UID: \"e5ed1052-5fa4-4d50-be77-d799d5722140\") " pod="openshift-multus/multus-d4598" Sep 12 01:44:59.827309 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.827217 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e5ed1052-5fa4-4d50-be77-d799d5722140-multus-socket-dir-parent\") pod \"multus-d4598\" (UID: \"e5ed1052-5fa4-4d50-be77-d799d5722140\") " pod="openshift-multus/multus-d4598" Sep 12 01:44:59.827309 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.827233 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/46256f89-0730-441e-97d5-6ef1dff74569-system-cni-dir\") pod \"multus-additional-cni-plugins-rzbs6\" (UID: \"46256f89-0730-441e-97d5-6ef1dff74569\") " pod="openshift-multus/multus-additional-cni-plugins-rzbs6" Sep 12 01:44:59.827309 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.827249 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/46256f89-0730-441e-97d5-6ef1dff74569-cni-binary-copy\") pod \"multus-additional-cni-plugins-rzbs6\" (UID: \"46256f89-0730-441e-97d5-6ef1dff74569\") " pod="openshift-multus/multus-additional-cni-plugins-rzbs6" Sep 12 01:44:59.827309 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.827266 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/46256f89-0730-441e-97d5-6ef1dff74569-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rzbs6\" (UID: \"46256f89-0730-441e-97d5-6ef1dff74569\") " pod="openshift-multus/multus-additional-cni-plugins-rzbs6" Sep 12 01:44:59.827309 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.827282 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbs9d\" (UniqueName: \"kubernetes.io/projected/46256f89-0730-441e-97d5-6ef1dff74569-kube-api-access-kbs9d\") pod \"multus-additional-cni-plugins-rzbs6\" (UID: \"46256f89-0730-441e-97d5-6ef1dff74569\") " pod="openshift-multus/multus-additional-cni-plugins-rzbs6" Sep 12 01:44:59.827810 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.827308 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/46256f89-0730-441e-97d5-6ef1dff74569-system-cni-dir\") pod \"multus-additional-cni-plugins-rzbs6\" (UID: \"46256f89-0730-441e-97d5-6ef1dff74569\") " pod="openshift-multus/multus-additional-cni-plugins-rzbs6" Sep 12 01:44:59.827810 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.827378 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5a68d964-2017-4185-b57a-5b09bfe19c4f-run\") pod \"tuned-4fhmh\" (UID: \"5a68d964-2017-4185-b57a-5b09bfe19c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4fhmh" Sep 12 01:44:59.827810 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.827422 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr2wt\" (UniqueName: \"kubernetes.io/projected/e7d6653c-acc4-4af4-89c4-39eff31ba128-kube-api-access-fr2wt\") pod \"aws-ebs-csi-driver-node-2shh6\" (UID: \"e7d6653c-acc4-4af4-89c4-39eff31ba128\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2shh6" Sep 12 01:44:59.827810 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.827466 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5ed1052-5fa4-4d50-be77-d799d5722140-host-var-lib-kubelet\") pod \"multus-d4598\" (UID: \"e5ed1052-5fa4-4d50-be77-d799d5722140\") " pod="openshift-multus/multus-d4598" Sep 12 01:44:59.827810 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.827490 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5a68d964-2017-4185-b57a-5b09bfe19c4f-etc-sysconfig\") pod \"tuned-4fhmh\" (UID: \"5a68d964-2017-4185-b57a-5b09bfe19c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4fhmh" Sep 12 01:44:59.827810 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.827513 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a68d964-2017-4185-b57a-5b09bfe19c4f-etc-kubernetes\") pod \"tuned-4fhmh\" (UID: \"5a68d964-2017-4185-b57a-5b09bfe19c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4fhmh" Sep 12 01:44:59.827810 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.827511 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5a68d964-2017-4185-b57a-5b09bfe19c4f-run\") pod \"tuned-4fhmh\" (UID: \"5a68d964-2017-4185-b57a-5b09bfe19c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4fhmh" Sep 12 01:44:59.827810 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.827533 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5a68d964-2017-4185-b57a-5b09bfe19c4f-etc-sysctl-d\") pod \"tuned-4fhmh\" (UID: \"5a68d964-2017-4185-b57a-5b09bfe19c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4fhmh" Sep 12 01:44:59.827810 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.827556 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/900016e2-2d16-467b-a924-83f1aec3610e-iptables-alerter-script\") pod \"iptables-alerter-pkjc5\" (UID: \"900016e2-2d16-467b-a924-83f1aec3610e\") " pod="openshift-network-operator/iptables-alerter-pkjc5" Sep 12 01:44:59.827810 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.827563 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5ed1052-5fa4-4d50-be77-d799d5722140-host-var-lib-kubelet\") pod \"multus-d4598\" (UID: \"e5ed1052-5fa4-4d50-be77-d799d5722140\") " pod="openshift-multus/multus-d4598" Sep 12 01:44:59.827810 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.827579 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e7d6653c-acc4-4af4-89c4-39eff31ba128-etc-selinux\") pod \"aws-ebs-csi-driver-node-2shh6\" (UID: \"e7d6653c-acc4-4af4-89c4-39eff31ba128\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2shh6" Sep 12 01:44:59.827810 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.827603 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/46256f89-0730-441e-97d5-6ef1dff74569-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rzbs6\" (UID: \"46256f89-0730-441e-97d5-6ef1dff74569\") " pod="openshift-multus/multus-additional-cni-plugins-rzbs6" Sep 12 01:44:59.827810 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.827627 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5ed1052-5fa4-4d50-be77-d799d5722140-multus-cni-dir\") pod \"multus-d4598\" (UID: \"e5ed1052-5fa4-4d50-be77-d799d5722140\") " pod="openshift-multus/multus-d4598" Sep 12 01:44:59.827810 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.827649 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e5ed1052-5fa4-4d50-be77-d799d5722140-cni-binary-copy\") pod \"multus-d4598\" (UID: \"e5ed1052-5fa4-4d50-be77-d799d5722140\") " pod="openshift-multus/multus-d4598" Sep 12 01:44:59.827810 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.827671 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmcgq\" (UniqueName: \"kubernetes.io/projected/e5ed1052-5fa4-4d50-be77-d799d5722140-kube-api-access-gmcgq\") pod \"multus-d4598\" (UID: \"e5ed1052-5fa4-4d50-be77-d799d5722140\") " pod="openshift-multus/multus-d4598" Sep 12 01:44:59.827810 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.827682 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a68d964-2017-4185-b57a-5b09bfe19c4f-etc-kubernetes\") pod \"tuned-4fhmh\" (UID: \"5a68d964-2017-4185-b57a-5b09bfe19c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4fhmh" Sep 12 01:44:59.827810 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.827695 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5a68d964-2017-4185-b57a-5b09bfe19c4f-etc-modprobe-d\") pod \"tuned-4fhmh\" (UID: \"5a68d964-2017-4185-b57a-5b09bfe19c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4fhmh" Sep 12 01:44:59.828670 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.827729 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5a68d964-2017-4185-b57a-5b09bfe19c4f-etc-sysconfig\") pod \"tuned-4fhmh\" (UID: \"5a68d964-2017-4185-b57a-5b09bfe19c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4fhmh" Sep 12 01:44:59.828670 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.827734 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e5ed1052-5fa4-4d50-be77-d799d5722140-cnibin\") pod \"multus-d4598\" (UID: \"e5ed1052-5fa4-4d50-be77-d799d5722140\") " pod="openshift-multus/multus-d4598" Sep 12 01:44:59.828670 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.827729 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e5ed1052-5fa4-4d50-be77-d799d5722140-multus-daemon-config\") pod \"multus-d4598\" (UID: \"e5ed1052-5fa4-4d50-be77-d799d5722140\") " pod="openshift-multus/multus-d4598" Sep 12 01:44:59.828670 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.827762 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e7d6653c-acc4-4af4-89c4-39eff31ba128-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2shh6\" (UID: \"e7d6653c-acc4-4af4-89c4-39eff31ba128\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2shh6" Sep 12 01:44:59.828670 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.827786 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e5ed1052-5fa4-4d50-be77-d799d5722140-host-var-lib-cni-bin\") pod \"multus-d4598\" (UID: \"e5ed1052-5fa4-4d50-be77-d799d5722140\") " pod="openshift-multus/multus-d4598" Sep 12 01:44:59.828670 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.827808 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e5ed1052-5fa4-4d50-be77-d799d5722140-hostroot\") pod \"multus-d4598\" (UID: \"e5ed1052-5fa4-4d50-be77-d799d5722140\") " pod="openshift-multus/multus-d4598" Sep 12 01:44:59.828670 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.827808 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/46256f89-0730-441e-97d5-6ef1dff74569-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rzbs6\" (UID: \"46256f89-0730-441e-97d5-6ef1dff74569\") " pod="openshift-multus/multus-additional-cni-plugins-rzbs6" Sep 12 01:44:59.828670 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.827826 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/46256f89-0730-441e-97d5-6ef1dff74569-cni-binary-copy\") pod \"multus-additional-cni-plugins-rzbs6\" (UID: \"46256f89-0730-441e-97d5-6ef1dff74569\") " pod="openshift-multus/multus-additional-cni-plugins-rzbs6" Sep 12 01:44:59.828670 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.827830 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a68d964-2017-4185-b57a-5b09bfe19c4f-var-lib-kubelet\") pod \"tuned-4fhmh\" (UID: \"5a68d964-2017-4185-b57a-5b09bfe19c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4fhmh" Sep 12 01:44:59.828670 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.827873 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5ed1052-5fa4-4d50-be77-d799d5722140-etc-kubernetes\") pod \"multus-d4598\" (UID: \"e5ed1052-5fa4-4d50-be77-d799d5722140\") " pod="openshift-multus/multus-d4598" Sep 12 01:44:59.828670 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.827878 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a68d964-2017-4185-b57a-5b09bfe19c4f-var-lib-kubelet\") pod \"tuned-4fhmh\" (UID: \"5a68d964-2017-4185-b57a-5b09bfe19c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4fhmh" Sep 12 01:44:59.828670 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.827894 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a68d964-2017-4185-b57a-5b09bfe19c4f-tmp\") pod \"tuned-4fhmh\" (UID: \"5a68d964-2017-4185-b57a-5b09bfe19c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4fhmh" Sep 12 01:44:59.828670 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.827919 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbzft\" (UniqueName: \"kubernetes.io/projected/900016e2-2d16-467b-a924-83f1aec3610e-kube-api-access-qbzft\") pod \"iptables-alerter-pkjc5\" (UID: \"900016e2-2d16-467b-a924-83f1aec3610e\") " pod="openshift-network-operator/iptables-alerter-pkjc5" Sep 12 01:44:59.828670 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.827928 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e7d6653c-acc4-4af4-89c4-39eff31ba128-etc-selinux\") pod \"aws-ebs-csi-driver-node-2shh6\" (UID: \"e7d6653c-acc4-4af4-89c4-39eff31ba128\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2shh6" Sep 12 01:44:59.828670 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.827949 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e5ed1052-5fa4-4d50-be77-d799d5722140-multus-conf-dir\") pod \"multus-d4598\" (UID: \"e5ed1052-5fa4-4d50-be77-d799d5722140\") " pod="openshift-multus/multus-d4598" Sep 12 01:44:59.828670 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.827973 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e5ed1052-5fa4-4d50-be77-d799d5722140-host-run-multus-certs\") pod \"multus-d4598\" (UID: \"e5ed1052-5fa4-4d50-be77-d799d5722140\") " pod="openshift-multus/multus-d4598" Sep 12 01:44:59.828670 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.827997 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b1408eab-ddcd-41d6-87af-50980dd43721-agent-certs\") pod \"konnectivity-agent-nhw89\" (UID: \"b1408eab-ddcd-41d6-87af-50980dd43721\") " pod="kube-system/konnectivity-agent-nhw89" Sep 12 01:44:59.829473 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.828022 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b1408eab-ddcd-41d6-87af-50980dd43721-konnectivity-ca\") pod \"konnectivity-agent-nhw89\" (UID: \"b1408eab-ddcd-41d6-87af-50980dd43721\") " pod="kube-system/konnectivity-agent-nhw89" Sep 12 01:44:59.829473 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.828031 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5a68d964-2017-4185-b57a-5b09bfe19c4f-etc-sysctl-d\") pod \"tuned-4fhmh\" (UID: \"5a68d964-2017-4185-b57a-5b09bfe19c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4fhmh" Sep 12 01:44:59.829473 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.828047 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5ed1052-5fa4-4d50-be77-d799d5722140-system-cni-dir\") pod \"multus-d4598\" (UID: \"e5ed1052-5fa4-4d50-be77-d799d5722140\") " pod="openshift-multus/multus-d4598" Sep 12 01:44:59.829473 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.828076 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e7d6653c-acc4-4af4-89c4-39eff31ba128-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2shh6\" (UID: \"e7d6653c-acc4-4af4-89c4-39eff31ba128\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2shh6" Sep 12 01:44:59.829473 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.828082 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e5ed1052-5fa4-4d50-be77-d799d5722140-host-var-lib-cni-multus\") pod \"multus-d4598\" (UID: \"e5ed1052-5fa4-4d50-be77-d799d5722140\") " pod="openshift-multus/multus-d4598" Sep 12 01:44:59.829473 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.828119 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/46256f89-0730-441e-97d5-6ef1dff74569-cnibin\") pod \"multus-additional-cni-plugins-rzbs6\" (UID: \"46256f89-0730-441e-97d5-6ef1dff74569\") " pod="openshift-multus/multus-additional-cni-plugins-rzbs6" Sep 12 01:44:59.829473 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.828126 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e5ed1052-5fa4-4d50-be77-d799d5722140-hostroot\") pod \"multus-d4598\" (UID: \"e5ed1052-5fa4-4d50-be77-d799d5722140\") " pod="openshift-multus/multus-d4598" Sep 12 01:44:59.829473 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.828146 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e5ed1052-5fa4-4d50-be77-d799d5722140-host-run-netns\") pod \"multus-d4598\" (UID: \"e5ed1052-5fa4-4d50-be77-d799d5722140\") " pod="openshift-multus/multus-d4598" Sep 12 01:44:59.829473 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.828169 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a68d964-2017-4185-b57a-5b09bfe19c4f-sys\") pod \"tuned-4fhmh\" (UID: \"5a68d964-2017-4185-b57a-5b09bfe19c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4fhmh" Sep 12 01:44:59.829473 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.828166 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e5ed1052-5fa4-4d50-be77-d799d5722140-cni-binary-copy\") pod \"multus-d4598\" (UID: \"e5ed1052-5fa4-4d50-be77-d799d5722140\") " pod="openshift-multus/multus-d4598" Sep 12 01:44:59.829473 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.828170 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e5ed1052-5fa4-4d50-be77-d799d5722140-host-var-lib-cni-bin\") pod \"multus-d4598\" (UID: \"e5ed1052-5fa4-4d50-be77-d799d5722140\") " pod="openshift-multus/multus-d4598" Sep 12 01:44:59.829473 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.828194 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e7d6653c-acc4-4af4-89c4-39eff31ba128-registration-dir\") pod \"aws-ebs-csi-driver-node-2shh6\" (UID: \"e7d6653c-acc4-4af4-89c4-39eff31ba128\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2shh6" Sep 12 01:44:59.829473 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.828200 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5ed1052-5fa4-4d50-be77-d799d5722140-etc-kubernetes\") pod \"multus-d4598\" (UID: \"e5ed1052-5fa4-4d50-be77-d799d5722140\") " pod="openshift-multus/multus-d4598" Sep 12 01:44:59.829473 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.828206 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/46256f89-0730-441e-97d5-6ef1dff74569-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rzbs6\" (UID: \"46256f89-0730-441e-97d5-6ef1dff74569\") " pod="openshift-multus/multus-additional-cni-plugins-rzbs6" Sep 12 01:44:59.829473 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.828224 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/900016e2-2d16-467b-a924-83f1aec3610e-iptables-alerter-script\") pod \"iptables-alerter-pkjc5\" (UID: \"900016e2-2d16-467b-a924-83f1aec3610e\") " pod="openshift-network-operator/iptables-alerter-pkjc5" Sep 12 01:44:59.829473 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.828246 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e5ed1052-5fa4-4d50-be77-d799d5722140-cnibin\") pod \"multus-d4598\" (UID: \"e5ed1052-5fa4-4d50-be77-d799d5722140\") " pod="openshift-multus/multus-d4598" Sep 12 01:44:59.829473 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.828255 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5ed1052-5fa4-4d50-be77-d799d5722140-system-cni-dir\") pod \"multus-d4598\" (UID: \"e5ed1052-5fa4-4d50-be77-d799d5722140\") " pod="openshift-multus/multus-d4598" Sep 12 01:44:59.829473 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.828267 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5ed1052-5fa4-4d50-be77-d799d5722140-multus-cni-dir\") pod \"multus-d4598\" (UID: \"e5ed1052-5fa4-4d50-be77-d799d5722140\") " pod="openshift-multus/multus-d4598" Sep 12 01:44:59.829995 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.828273 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e5ed1052-5fa4-4d50-be77-d799d5722140-multus-conf-dir\") pod \"multus-d4598\" (UID: \"e5ed1052-5fa4-4d50-be77-d799d5722140\") " pod="openshift-multus/multus-d4598" Sep 12 01:44:59.829995 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.828281 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e5ed1052-5fa4-4d50-be77-d799d5722140-host-run-multus-certs\") pod \"multus-d4598\" (UID: \"e5ed1052-5fa4-4d50-be77-d799d5722140\") " pod="openshift-multus/multus-d4598" Sep 12 01:44:59.829995 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.828168 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5a68d964-2017-4185-b57a-5b09bfe19c4f-etc-modprobe-d\") pod \"tuned-4fhmh\" (UID: \"5a68d964-2017-4185-b57a-5b09bfe19c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4fhmh" Sep 12 01:44:59.829995 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.828312 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e5ed1052-5fa4-4d50-be77-d799d5722140-host-var-lib-cni-multus\") pod \"multus-d4598\" (UID: \"e5ed1052-5fa4-4d50-be77-d799d5722140\") " pod="openshift-multus/multus-d4598" Sep 12 01:44:59.829995 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.828349 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e5ed1052-5fa4-4d50-be77-d799d5722140-host-run-netns\") pod \"multus-d4598\" (UID: \"e5ed1052-5fa4-4d50-be77-d799d5722140\") " pod="openshift-multus/multus-d4598" Sep 12 01:44:59.829995 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.828386 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/46256f89-0730-441e-97d5-6ef1dff74569-cnibin\") pod \"multus-additional-cni-plugins-rzbs6\" (UID: \"46256f89-0730-441e-97d5-6ef1dff74569\") " pod="openshift-multus/multus-additional-cni-plugins-rzbs6" Sep 12 01:44:59.829995 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.828393 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e7d6653c-acc4-4af4-89c4-39eff31ba128-registration-dir\") pod \"aws-ebs-csi-driver-node-2shh6\" (UID: \"e7d6653c-acc4-4af4-89c4-39eff31ba128\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2shh6" Sep 12 01:44:59.829995 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.828420 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a68d964-2017-4185-b57a-5b09bfe19c4f-sys\") pod \"tuned-4fhmh\" (UID: \"5a68d964-2017-4185-b57a-5b09bfe19c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4fhmh" Sep 12 01:44:59.829995 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.828467 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/46256f89-0730-441e-97d5-6ef1dff74569-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rzbs6\" (UID: \"46256f89-0730-441e-97d5-6ef1dff74569\") " pod="openshift-multus/multus-additional-cni-plugins-rzbs6" Sep 12 01:44:59.829995 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.828501 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5a68d964-2017-4185-b57a-5b09bfe19c4f-etc-tuned\") pod \"tuned-4fhmh\" (UID: \"5a68d964-2017-4185-b57a-5b09bfe19c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4fhmh" Sep 12 01:44:59.829995 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.828526 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/900016e2-2d16-467b-a924-83f1aec3610e-host-slash\") pod \"iptables-alerter-pkjc5\" (UID: \"900016e2-2d16-467b-a924-83f1aec3610e\") " pod="openshift-network-operator/iptables-alerter-pkjc5" Sep 12 01:44:59.829995 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.828547 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5a68d964-2017-4185-b57a-5b09bfe19c4f-etc-sysctl-conf\") pod \"tuned-4fhmh\" (UID: \"5a68d964-2017-4185-b57a-5b09bfe19c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4fhmh" Sep 12 01:44:59.829995 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.828597 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e5ed1052-5fa4-4d50-be77-d799d5722140-host-run-k8s-cni-cncf-io\") pod \"multus-d4598\" (UID: \"e5ed1052-5fa4-4d50-be77-d799d5722140\") " pod="openshift-multus/multus-d4598" Sep 12 01:44:59.829995 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.828618 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5a68d964-2017-4185-b57a-5b09bfe19c4f-lib-modules\") pod \"tuned-4fhmh\" (UID: \"5a68d964-2017-4185-b57a-5b09bfe19c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4fhmh" Sep 12 01:44:59.829995 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.828631 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/46256f89-0730-441e-97d5-6ef1dff74569-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rzbs6\" (UID: \"46256f89-0730-441e-97d5-6ef1dff74569\") " pod="openshift-multus/multus-additional-cni-plugins-rzbs6" Sep 12 01:44:59.829995 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.828749 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5a68d964-2017-4185-b57a-5b09bfe19c4f-etc-sysctl-conf\") pod \"tuned-4fhmh\" (UID: \"5a68d964-2017-4185-b57a-5b09bfe19c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4fhmh" Sep 12 01:44:59.829995 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.828641 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-994lz\" (UniqueName: \"kubernetes.io/projected/5a68d964-2017-4185-b57a-5b09bfe19c4f-kube-api-access-994lz\") pod \"tuned-4fhmh\" (UID: \"5a68d964-2017-4185-b57a-5b09bfe19c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4fhmh" Sep 12 01:44:59.830467 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.828795 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e7d6653c-acc4-4af4-89c4-39eff31ba128-socket-dir\") pod \"aws-ebs-csi-driver-node-2shh6\" (UID: \"e7d6653c-acc4-4af4-89c4-39eff31ba128\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2shh6" Sep 12 01:44:59.830467 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.828822 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5a68d964-2017-4185-b57a-5b09bfe19c4f-etc-systemd\") pod \"tuned-4fhmh\" (UID: \"5a68d964-2017-4185-b57a-5b09bfe19c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4fhmh" Sep 12 01:44:59.830467 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.828840 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a68d964-2017-4185-b57a-5b09bfe19c4f-host\") pod \"tuned-4fhmh\" (UID: \"5a68d964-2017-4185-b57a-5b09bfe19c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4fhmh" Sep 12 01:44:59.830467 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.828898 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a68d964-2017-4185-b57a-5b09bfe19c4f-host\") pod \"tuned-4fhmh\" (UID: \"5a68d964-2017-4185-b57a-5b09bfe19c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4fhmh" Sep 12 01:44:59.830467 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.828894 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/900016e2-2d16-467b-a924-83f1aec3610e-host-slash\") pod \"iptables-alerter-pkjc5\" (UID: \"900016e2-2d16-467b-a924-83f1aec3610e\") " pod="openshift-network-operator/iptables-alerter-pkjc5" Sep 12 01:44:59.830467 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.828933 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e5ed1052-5fa4-4d50-be77-d799d5722140-host-run-k8s-cni-cncf-io\") pod \"multus-d4598\" (UID: \"e5ed1052-5fa4-4d50-be77-d799d5722140\") " pod="openshift-multus/multus-d4598" Sep 12 01:44:59.830467 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.828943 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b1408eab-ddcd-41d6-87af-50980dd43721-konnectivity-ca\") pod \"konnectivity-agent-nhw89\" (UID: \"b1408eab-ddcd-41d6-87af-50980dd43721\") " pod="kube-system/konnectivity-agent-nhw89" Sep 12 01:44:59.830467 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.829056 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e7d6653c-acc4-4af4-89c4-39eff31ba128-socket-dir\") pod \"aws-ebs-csi-driver-node-2shh6\" (UID: \"e7d6653c-acc4-4af4-89c4-39eff31ba128\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2shh6" Sep 12 01:44:59.830467 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.829089 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5a68d964-2017-4185-b57a-5b09bfe19c4f-etc-systemd\") pod \"tuned-4fhmh\" (UID: \"5a68d964-2017-4185-b57a-5b09bfe19c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4fhmh" Sep 12 01:44:59.830467 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.829163 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5a68d964-2017-4185-b57a-5b09bfe19c4f-lib-modules\") pod \"tuned-4fhmh\" (UID: \"5a68d964-2017-4185-b57a-5b09bfe19c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4fhmh" Sep 12 01:44:59.830943 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.830904 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5a68d964-2017-4185-b57a-5b09bfe19c4f-etc-tuned\") pod \"tuned-4fhmh\" (UID: \"5a68d964-2017-4185-b57a-5b09bfe19c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4fhmh" Sep 12 01:44:59.831075 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.831053 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a68d964-2017-4185-b57a-5b09bfe19c4f-tmp\") pod \"tuned-4fhmh\" (UID: \"5a68d964-2017-4185-b57a-5b09bfe19c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4fhmh" Sep 12 01:44:59.831223 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.831209 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b1408eab-ddcd-41d6-87af-50980dd43721-agent-certs\") pod \"konnectivity-agent-nhw89\" (UID: \"b1408eab-ddcd-41d6-87af-50980dd43721\") " pod="kube-system/konnectivity-agent-nhw89" Sep 12 01:44:59.835513 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.835493 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmcgq\" (UniqueName: \"kubernetes.io/projected/e5ed1052-5fa4-4d50-be77-d799d5722140-kube-api-access-gmcgq\") pod \"multus-d4598\" (UID: \"e5ed1052-5fa4-4d50-be77-d799d5722140\") " pod="openshift-multus/multus-d4598" Sep 12 01:44:59.835776 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.835758 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr2wt\" (UniqueName: \"kubernetes.io/projected/e7d6653c-acc4-4af4-89c4-39eff31ba128-kube-api-access-fr2wt\") pod \"aws-ebs-csi-driver-node-2shh6\" (UID: \"e7d6653c-acc4-4af4-89c4-39eff31ba128\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2shh6" Sep 12 01:44:59.836714 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.836690 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-994lz\" (UniqueName: \"kubernetes.io/projected/5a68d964-2017-4185-b57a-5b09bfe19c4f-kube-api-access-994lz\") pod \"tuned-4fhmh\" (UID: \"5a68d964-2017-4185-b57a-5b09bfe19c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4fhmh" Sep 12 01:44:59.836988 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.836969 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbzft\" (UniqueName: \"kubernetes.io/projected/900016e2-2d16-467b-a924-83f1aec3610e-kube-api-access-qbzft\") pod \"iptables-alerter-pkjc5\" (UID: \"900016e2-2d16-467b-a924-83f1aec3610e\") " pod="openshift-network-operator/iptables-alerter-pkjc5" Sep 12 01:44:59.839060 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.839010 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbs9d\" (UniqueName: \"kubernetes.io/projected/46256f89-0730-441e-97d5-6ef1dff74569-kube-api-access-kbs9d\") pod \"multus-additional-cni-plugins-rzbs6\" (UID: \"46256f89-0730-441e-97d5-6ef1dff74569\") " pod="openshift-multus/multus-additional-cni-plugins-rzbs6" Sep 12 01:44:59.916072 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.916030 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qzpw5" Sep 12 01:44:59.924991 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.924963 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2dpmw" Sep 12 01:44:59.933535 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.933512 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:44:59.939214 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.939195 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4fhmh" Sep 12 01:44:59.945746 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.945719 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pkjc5" Sep 12 01:44:59.952226 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.952206 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2shh6" Sep 12 01:44:59.958763 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.958739 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nhw89" Sep 12 01:44:59.965410 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.965394 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rzbs6" Sep 12 01:44:59.970915 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:44:59.970898 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-d4598" Sep 12 01:45:00.231688 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:00.231592 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frwnn\" (UniqueName: \"kubernetes.io/projected/f02f888c-7971-4e66-8a7d-3e180b6807d9-kube-api-access-frwnn\") pod \"network-check-target-bxnbn\" (UID: \"f02f888c-7971-4e66-8a7d-3e180b6807d9\") " pod="openshift-network-diagnostics/network-check-target-bxnbn" Sep 12 01:45:00.231688 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:00.231656 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03-metrics-certs\") pod \"network-metrics-daemon-nhq7s\" (UID: \"b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03\") " pod="openshift-multus/network-metrics-daemon-nhq7s" Sep 12 01:45:00.231912 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:00.231691 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f016f46e-b786-405e-8f6e-faca9e639b5f-original-pull-secret\") pod \"global-pull-secret-syncer-wjmgj\" (UID: \"f016f46e-b786-405e-8f6e-faca9e639b5f\") " pod="kube-system/global-pull-secret-syncer-wjmgj" Sep 12 01:45:00.231912 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:00.231789 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 12 01:45:00.231912 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:00.231806 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 12 01:45:00.231912 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:00.231808 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 12 01:45:00.231912 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:00.231810 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 12 01:45:00.231912 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:00.231866 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f016f46e-b786-405e-8f6e-faca9e639b5f-original-pull-secret podName:f016f46e-b786-405e-8f6e-faca9e639b5f nodeName:}" failed. No retries permitted until 2026-09-12 01:45:01.231848805 +0000 UTC m=+4.023842593 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f016f46e-b786-405e-8f6e-faca9e639b5f-original-pull-secret") pod "global-pull-secret-syncer-wjmgj" (UID: "f016f46e-b786-405e-8f6e-faca9e639b5f") : object "kube-system"/"original-pull-secret" not registered Sep 12 01:45:00.231912 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:00.231877 2579 projected.go:196] Error preparing data for projected volume kube-api-access-frwnn for pod openshift-network-diagnostics/network-check-target-bxnbn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 12 01:45:00.231912 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:00.231881 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03-metrics-certs podName:b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03 nodeName:}" failed. No retries permitted until 2026-09-12 01:45:01.231874689 +0000 UTC m=+4.023868462 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03-metrics-certs") pod "network-metrics-daemon-nhq7s" (UID: "b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 12 01:45:00.232264 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:00.231939 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f02f888c-7971-4e66-8a7d-3e180b6807d9-kube-api-access-frwnn podName:f02f888c-7971-4e66-8a7d-3e180b6807d9 nodeName:}" failed. No retries permitted until 2026-09-12 01:45:01.231920102 +0000 UTC m=+4.023913889 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-frwnn" (UniqueName: "kubernetes.io/projected/f02f888c-7971-4e66-8a7d-3e180b6807d9-kube-api-access-frwnn") pod "network-check-target-bxnbn" (UID: "f02f888c-7971-4e66-8a7d-3e180b6807d9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 12 01:45:00.381700 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:00.381673 2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 12 01:45:00.568022 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:45:00.567989 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7d6653c_acc4_4af4_89c4_39eff31ba128.slice/crio-75430eaf11f1936852cb2c1806eb42badb52509ecfb94983ffe848bd7384ee5b WatchSource:0}: Error finding container 75430eaf11f1936852cb2c1806eb42badb52509ecfb94983ffe848bd7384ee5b: Status 404 returned error can't find the container with id 75430eaf11f1936852cb2c1806eb42badb52509ecfb94983ffe848bd7384ee5b Sep 12 01:45:00.568920 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:45:00.568896 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e1eb253_a93e_4370_9cbd_a5ec2ae013a9.slice/crio-416b215b4c42fba1e9ed287d577ca66f85d9bcf355dd9cbdb2df876231e8ca39 WatchSource:0}: Error finding container 416b215b4c42fba1e9ed287d577ca66f85d9bcf355dd9cbdb2df876231e8ca39: Status 404 returned error can't find the container with id 416b215b4c42fba1e9ed287d577ca66f85d9bcf355dd9cbdb2df876231e8ca39 Sep 12 01:45:00.570084 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:45:00.570063 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46256f89_0730_441e_97d5_6ef1dff74569.slice/crio-ed0640640d13650b7f28626d218300a468e1e315c8e6b83235530a2939eab8e9 WatchSource:0}: Error finding container ed0640640d13650b7f28626d218300a468e1e315c8e6b83235530a2939eab8e9: Status 404 returned error can't find the container with id ed0640640d13650b7f28626d218300a468e1e315c8e6b83235530a2939eab8e9 Sep 12 01:45:00.572400 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:45:00.571783 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9274dec2_bf50_42c6_8c65_7ecb88771cc7.slice/crio-551ef37714e287a9c8b5be8dce209193bcafef919157267513396a6116296288 WatchSource:0}: Error finding container 551ef37714e287a9c8b5be8dce209193bcafef919157267513396a6116296288: Status 404 returned error can't find the container with id 551ef37714e287a9c8b5be8dce209193bcafef919157267513396a6116296288 Sep 12 01:45:00.576563 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:45:00.576540 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod900016e2_2d16_467b_a924_83f1aec3610e.slice/crio-85ab3e0747c8c7fa73e1e2249901df79dba88a1273cf221a4d5caa631b758931 WatchSource:0}: Error finding container 85ab3e0747c8c7fa73e1e2249901df79dba88a1273cf221a4d5caa631b758931: Status 404 returned error can't find the container with id 85ab3e0747c8c7fa73e1e2249901df79dba88a1273cf221a4d5caa631b758931 Sep 12 01:45:00.576759 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:45:00.576719 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1408eab_ddcd_41d6_87af_50980dd43721.slice/crio-656d0a45986f5a430e91f548f4abae6b8a365959eaa1a5cb42b2af120ed39f3f WatchSource:0}: Error finding container 656d0a45986f5a430e91f548f4abae6b8a365959eaa1a5cb42b2af120ed39f3f: Status 404 returned error can't find the container with id 656d0a45986f5a430e91f548f4abae6b8a365959eaa1a5cb42b2af120ed39f3f Sep 12 01:45:00.578016 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:45:00.577984 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1140d6e_61e4_4bad_a337_cc28db7dd812.slice/crio-e01d4a0559cb88644bc10c4fbe03154c55fba698eb06f918fde8df5b976366a7 WatchSource:0}: Error finding container e01d4a0559cb88644bc10c4fbe03154c55fba698eb06f918fde8df5b976366a7: Status 404 returned error can't find the container with id e01d4a0559cb88644bc10c4fbe03154c55fba698eb06f918fde8df5b976366a7 Sep 12 01:45:00.578722 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:45:00.578699 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5ed1052_5fa4_4d50_be77_d799d5722140.slice/crio-c866293db16f4f3255e606e5ccedcd06ec51537e200e32eb2f2a3c7b85664c9c WatchSource:0}: Error finding container c866293db16f4f3255e606e5ccedcd06ec51537e200e32eb2f2a3c7b85664c9c: Status 404 returned error can't find the container with id c866293db16f4f3255e606e5ccedcd06ec51537e200e32eb2f2a3c7b85664c9c Sep 12 01:45:00.580501 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:45:00.580478 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a68d964_2017_4185_b57a_5b09bfe19c4f.slice/crio-2cff46c1b533bb2b26f5d465fdb9625bcca0d4dea52d0d1089404fb02e124b5c WatchSource:0}: Error finding container 2cff46c1b533bb2b26f5d465fdb9625bcca0d4dea52d0d1089404fb02e124b5c: Status 404 returned error can't find the container with id 2cff46c1b533bb2b26f5d465fdb9625bcca0d4dea52d0d1089404fb02e124b5c Sep 12 01:45:00.655436 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:00.655394 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-11 01:39:59 +0000 UTC" deadline="2028-04-24 09:44:39.994740784 +0000 UTC" Sep 12 01:45:00.655436 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:00.655436 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14167h59m39.33930892s" Sep 12 01:45:00.729433 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:00.729408 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4fhmh" event={"ID":"5a68d964-2017-4185-b57a-5b09bfe19c4f","Type":"ContainerStarted","Data":"2cff46c1b533bb2b26f5d465fdb9625bcca0d4dea52d0d1089404fb02e124b5c"} Sep 12 01:45:00.731016 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:00.730631 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pkjc5" event={"ID":"900016e2-2d16-467b-a924-83f1aec3610e","Type":"ContainerStarted","Data":"85ab3e0747c8c7fa73e1e2249901df79dba88a1273cf221a4d5caa631b758931"} Sep 12 01:45:00.734425 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:00.734396 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qzpw5" event={"ID":"a1140d6e-61e4-4bad-a337-cc28db7dd812","Type":"ContainerStarted","Data":"e01d4a0559cb88644bc10c4fbe03154c55fba698eb06f918fde8df5b976366a7"} Sep 12 01:45:00.735497 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:00.735472 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" event={"ID":"9274dec2-bf50-42c6-8c65-7ecb88771cc7","Type":"ContainerStarted","Data":"551ef37714e287a9c8b5be8dce209193bcafef919157267513396a6116296288"} Sep 12 01:45:00.736614 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:00.736593 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2dpmw" event={"ID":"6e1eb253-a93e-4370-9cbd-a5ec2ae013a9","Type":"ContainerStarted","Data":"416b215b4c42fba1e9ed287d577ca66f85d9bcf355dd9cbdb2df876231e8ca39"} Sep 12 01:45:00.737652 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:00.737633 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rzbs6" event={"ID":"46256f89-0730-441e-97d5-6ef1dff74569","Type":"ContainerStarted","Data":"ed0640640d13650b7f28626d218300a468e1e315c8e6b83235530a2939eab8e9"} Sep 12 01:45:00.738742 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:00.738722 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2shh6" event={"ID":"e7d6653c-acc4-4af4-89c4-39eff31ba128","Type":"ContainerStarted","Data":"75430eaf11f1936852cb2c1806eb42badb52509ecfb94983ffe848bd7384ee5b"} Sep 12 01:45:00.739913 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:00.739889 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-d4598" event={"ID":"e5ed1052-5fa4-4d50-be77-d799d5722140","Type":"ContainerStarted","Data":"c866293db16f4f3255e606e5ccedcd06ec51537e200e32eb2f2a3c7b85664c9c"} Sep 12 01:45:00.741008 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:00.740983 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nhw89" event={"ID":"b1408eab-ddcd-41d6-87af-50980dd43721","Type":"ContainerStarted","Data":"656d0a45986f5a430e91f548f4abae6b8a365959eaa1a5cb42b2af120ed39f3f"} Sep 12 01:45:00.742783 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:00.742765 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-79.ec2.internal" event={"ID":"0f092973866d00f19dd94cc0896a8414","Type":"ContainerStarted","Data":"ecbfe5d7bc6562ebede03421e654193e78fcee2468fd35c9bd1c1471eaf27238"} Sep 12 01:45:00.759419 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:00.759340 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-79.ec2.internal" podStartSLOduration=2.7593221310000002 podStartE2EDuration="2.759322131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 01:44:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:45:00.758418266 +0000 UTC m=+3.550412061" watchObservedRunningTime="2026-09-12 01:45:00.759322131 +0000 UTC m=+3.551315926" Sep 12 01:45:01.238660 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:01.238622 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03-metrics-certs\") pod \"network-metrics-daemon-nhq7s\" (UID: \"b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03\") " pod="openshift-multus/network-metrics-daemon-nhq7s" Sep 12 01:45:01.238824 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:01.238686 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f016f46e-b786-405e-8f6e-faca9e639b5f-original-pull-secret\") pod \"global-pull-secret-syncer-wjmgj\" (UID: \"f016f46e-b786-405e-8f6e-faca9e639b5f\") " pod="kube-system/global-pull-secret-syncer-wjmgj" Sep 12 01:45:01.238824 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:01.238755 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frwnn\" (UniqueName: \"kubernetes.io/projected/f02f888c-7971-4e66-8a7d-3e180b6807d9-kube-api-access-frwnn\") pod \"network-check-target-bxnbn\" (UID: \"f02f888c-7971-4e66-8a7d-3e180b6807d9\") " pod="openshift-network-diagnostics/network-check-target-bxnbn" Sep 12 01:45:01.238935 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:01.238899 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 12 01:45:01.238935 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:01.238922 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 12 01:45:01.238935 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:01.238934 2579 projected.go:196] Error preparing data for projected volume kube-api-access-frwnn for pod openshift-network-diagnostics/network-check-target-bxnbn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 12 01:45:01.239085 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:01.238991 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f02f888c-7971-4e66-8a7d-3e180b6807d9-kube-api-access-frwnn podName:f02f888c-7971-4e66-8a7d-3e180b6807d9 nodeName:}" failed. No retries permitted until 2026-09-12 01:45:03.238973708 +0000 UTC m=+6.030967494 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-frwnn" (UniqueName: "kubernetes.io/projected/f02f888c-7971-4e66-8a7d-3e180b6807d9-kube-api-access-frwnn") pod "network-check-target-bxnbn" (UID: "f02f888c-7971-4e66-8a7d-3e180b6807d9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 12 01:45:01.239382 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:01.239363 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 12 01:45:01.239476 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:01.239417 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03-metrics-certs podName:b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03 nodeName:}" failed. No retries permitted until 2026-09-12 01:45:03.239401948 +0000 UTC m=+6.031395933 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03-metrics-certs") pod "network-metrics-daemon-nhq7s" (UID: "b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 12 01:45:01.239543 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:01.239487 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 12 01:45:01.239543 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:01.239521 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f016f46e-b786-405e-8f6e-faca9e639b5f-original-pull-secret podName:f016f46e-b786-405e-8f6e-faca9e639b5f nodeName:}" failed. No retries permitted until 2026-09-12 01:45:03.239510642 +0000 UTC m=+6.031504429 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f016f46e-b786-405e-8f6e-faca9e639b5f-original-pull-secret") pod "global-pull-secret-syncer-wjmgj" (UID: "f016f46e-b786-405e-8f6e-faca9e639b5f") : object "kube-system"/"original-pull-secret" not registered Sep 12 01:45:01.656239 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:01.656135 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-11 01:39:59 +0000 UTC" deadline="2028-04-15 16:08:25.189070644 +0000 UTC" Sep 12 01:45:01.656239 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:01.656196 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13958h23m23.532879071s" Sep 12 01:45:01.722209 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:01.722176 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bxnbn" Sep 12 01:45:01.722363 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:01.722307 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bxnbn" podUID="f02f888c-7971-4e66-8a7d-3e180b6807d9" Sep 12 01:45:01.722724 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:01.722705 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wjmgj" Sep 12 01:45:01.722821 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:01.722801 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wjmgj" podUID="f016f46e-b786-405e-8f6e-faca9e639b5f" Sep 12 01:45:01.722884 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:01.722871 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhq7s" Sep 12 01:45:01.722967 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:01.722948 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhq7s" podUID="b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03" Sep 12 01:45:01.757232 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:01.757197 2579 generic.go:356] "Generic (PLEG): container finished" podID="336bcc6551e46b69eadff66cca9f19f9" containerID="0fa68897f7120dc0d74033c8bcb31019e545df78560acc2327ae3d0409ae6ebb" exitCode=0 Sep 12 01:45:01.758152 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:01.758127 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-79.ec2.internal" event={"ID":"336bcc6551e46b69eadff66cca9f19f9","Type":"ContainerDied","Data":"0fa68897f7120dc0d74033c8bcb31019e545df78560acc2327ae3d0409ae6ebb"} Sep 12 01:45:02.779591 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:02.779439 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-79.ec2.internal" event={"ID":"336bcc6551e46b69eadff66cca9f19f9","Type":"ContainerStarted","Data":"e6b225e513f4df7b2de87c2a571d54a92fd7442b87b9d87410417f4983b3757d"} Sep 12 01:45:03.257085 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:03.257001 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frwnn\" (UniqueName: \"kubernetes.io/projected/f02f888c-7971-4e66-8a7d-3e180b6807d9-kube-api-access-frwnn\") pod \"network-check-target-bxnbn\" (UID: \"f02f888c-7971-4e66-8a7d-3e180b6807d9\") " pod="openshift-network-diagnostics/network-check-target-bxnbn" Sep 12 01:45:03.257247 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:03.257124 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03-metrics-certs\") pod \"network-metrics-daemon-nhq7s\" (UID: \"b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03\") " pod="openshift-multus/network-metrics-daemon-nhq7s" Sep 12 01:45:03.257247 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:03.257169 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f016f46e-b786-405e-8f6e-faca9e639b5f-original-pull-secret\") pod \"global-pull-secret-syncer-wjmgj\" (UID: \"f016f46e-b786-405e-8f6e-faca9e639b5f\") " pod="kube-system/global-pull-secret-syncer-wjmgj" Sep 12 01:45:03.257353 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:03.257325 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 12 01:45:03.257405 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:03.257391 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f016f46e-b786-405e-8f6e-faca9e639b5f-original-pull-secret podName:f016f46e-b786-405e-8f6e-faca9e639b5f nodeName:}" failed. No retries permitted until 2026-09-12 01:45:07.257373235 +0000 UTC m=+10.049367021 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f016f46e-b786-405e-8f6e-faca9e639b5f-original-pull-secret") pod "global-pull-secret-syncer-wjmgj" (UID: "f016f46e-b786-405e-8f6e-faca9e639b5f") : object "kube-system"/"original-pull-secret" not registered Sep 12 01:45:03.257821 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:03.257801 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 12 01:45:03.257903 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:03.257827 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 12 01:45:03.257903 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:03.257839 2579 projected.go:196] Error preparing data for projected volume kube-api-access-frwnn for pod openshift-network-diagnostics/network-check-target-bxnbn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 12 01:45:03.257903 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:03.257882 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f02f888c-7971-4e66-8a7d-3e180b6807d9-kube-api-access-frwnn podName:f02f888c-7971-4e66-8a7d-3e180b6807d9 nodeName:}" failed. No retries permitted until 2026-09-12 01:45:07.257868148 +0000 UTC m=+10.049861931 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-frwnn" (UniqueName: "kubernetes.io/projected/f02f888c-7971-4e66-8a7d-3e180b6807d9-kube-api-access-frwnn") pod "network-check-target-bxnbn" (UID: "f02f888c-7971-4e66-8a7d-3e180b6807d9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 12 01:45:03.258057 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:03.257946 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 12 01:45:03.258057 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:03.257979 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03-metrics-certs podName:b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03 nodeName:}" failed. No retries permitted until 2026-09-12 01:45:07.257969116 +0000 UTC m=+10.049962896 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03-metrics-certs") pod "network-metrics-daemon-nhq7s" (UID: "b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 12 01:45:03.720689 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:03.720617 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhq7s" Sep 12 01:45:03.720689 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:03.720678 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bxnbn" Sep 12 01:45:03.720895 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:03.720784 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bxnbn" podUID="f02f888c-7971-4e66-8a7d-3e180b6807d9" Sep 12 01:45:03.721158 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:03.721136 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wjmgj" Sep 12 01:45:03.721268 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:03.721236 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wjmgj" podUID="f016f46e-b786-405e-8f6e-faca9e639b5f" Sep 12 01:45:03.723321 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:03.723291 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhq7s" podUID="b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03" Sep 12 01:45:05.720605 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:05.720573 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wjmgj" Sep 12 01:45:05.721062 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:05.720703 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wjmgj" podUID="f016f46e-b786-405e-8f6e-faca9e639b5f" Sep 12 01:45:05.721206 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:05.721121 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhq7s" Sep 12 01:45:05.721281 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:05.721220 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhq7s" podUID="b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03" Sep 12 01:45:05.721404 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:05.721383 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bxnbn" Sep 12 01:45:05.721516 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:05.721495 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bxnbn" podUID="f02f888c-7971-4e66-8a7d-3e180b6807d9" Sep 12 01:45:07.296137 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:07.295825 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frwnn\" (UniqueName: \"kubernetes.io/projected/f02f888c-7971-4e66-8a7d-3e180b6807d9-kube-api-access-frwnn\") pod \"network-check-target-bxnbn\" (UID: \"f02f888c-7971-4e66-8a7d-3e180b6807d9\") " pod="openshift-network-diagnostics/network-check-target-bxnbn" Sep 12 01:45:07.296553 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:07.296166 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03-metrics-certs\") pod \"network-metrics-daemon-nhq7s\" (UID: \"b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03\") " pod="openshift-multus/network-metrics-daemon-nhq7s" Sep 12 01:45:07.296553 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:07.296017 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 12 01:45:07.296553 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:07.296242 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 12 01:45:07.296553 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:07.296260 2579 projected.go:196] Error preparing data for projected volume kube-api-access-frwnn for pod openshift-network-diagnostics/network-check-target-bxnbn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 12 01:45:07.296553 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:07.296294 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 12 01:45:07.296553 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:07.296323 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f02f888c-7971-4e66-8a7d-3e180b6807d9-kube-api-access-frwnn podName:f02f888c-7971-4e66-8a7d-3e180b6807d9 nodeName:}" failed. No retries permitted until 2026-09-12 01:45:15.296301931 +0000 UTC m=+18.088295723 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-frwnn" (UniqueName: "kubernetes.io/projected/f02f888c-7971-4e66-8a7d-3e180b6807d9-kube-api-access-frwnn") pod "network-check-target-bxnbn" (UID: "f02f888c-7971-4e66-8a7d-3e180b6807d9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 12 01:45:07.296553 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:07.296348 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f016f46e-b786-405e-8f6e-faca9e639b5f-original-pull-secret podName:f016f46e-b786-405e-8f6e-faca9e639b5f nodeName:}" failed. No retries permitted until 2026-09-12 01:45:15.296336157 +0000 UTC m=+18.088329948 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f016f46e-b786-405e-8f6e-faca9e639b5f-original-pull-secret") pod "global-pull-secret-syncer-wjmgj" (UID: "f016f46e-b786-405e-8f6e-faca9e639b5f") : object "kube-system"/"original-pull-secret" not registered Sep 12 01:45:07.296553 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:07.296210 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f016f46e-b786-405e-8f6e-faca9e639b5f-original-pull-secret\") pod \"global-pull-secret-syncer-wjmgj\" (UID: \"f016f46e-b786-405e-8f6e-faca9e639b5f\") " pod="kube-system/global-pull-secret-syncer-wjmgj" Sep 12 01:45:07.296553 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:07.296372 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 12 01:45:07.296553 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:07.296407 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03-metrics-certs podName:b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03 nodeName:}" failed. No retries permitted until 2026-09-12 01:45:15.296391622 +0000 UTC m=+18.088385411 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03-metrics-certs") pod "network-metrics-daemon-nhq7s" (UID: "b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 12 01:45:07.721342 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:07.721307 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhq7s" Sep 12 01:45:07.721522 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:07.721423 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bxnbn" Sep 12 01:45:07.721522 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:07.721476 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhq7s" podUID="b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03" Sep 12 01:45:07.721652 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:07.721517 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bxnbn" podUID="f02f888c-7971-4e66-8a7d-3e180b6807d9" Sep 12 01:45:07.721652 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:07.721556 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wjmgj" Sep 12 01:45:07.721652 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:07.721632 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wjmgj" podUID="f016f46e-b786-405e-8f6e-faca9e639b5f" Sep 12 01:45:09.720917 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:09.720434 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bxnbn" Sep 12 01:45:09.720917 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:09.720437 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wjmgj" Sep 12 01:45:09.720917 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:09.720591 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhq7s" Sep 12 01:45:09.720917 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:09.720581 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bxnbn" podUID="f02f888c-7971-4e66-8a7d-3e180b6807d9" Sep 12 01:45:09.720917 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:09.720687 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wjmgj" podUID="f016f46e-b786-405e-8f6e-faca9e639b5f" Sep 12 01:45:09.720917 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:09.720766 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhq7s" podUID="b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03" Sep 12 01:45:11.719980 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:11.719939 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wjmgj" Sep 12 01:45:11.720490 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:11.720079 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wjmgj" podUID="f016f46e-b786-405e-8f6e-faca9e639b5f" Sep 12 01:45:11.720490 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:11.720424 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhq7s" Sep 12 01:45:11.720618 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:11.720526 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhq7s" podUID="b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03" Sep 12 01:45:11.720676 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:11.720635 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bxnbn" Sep 12 01:45:11.720725 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:11.720692 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bxnbn" podUID="f02f888c-7971-4e66-8a7d-3e180b6807d9" Sep 12 01:45:13.720057 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:13.720024 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wjmgj" Sep 12 01:45:13.720564 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:13.720031 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhq7s" Sep 12 01:45:13.720564 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:13.720144 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wjmgj" podUID="f016f46e-b786-405e-8f6e-faca9e639b5f" Sep 12 01:45:13.720564 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:13.720160 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bxnbn" Sep 12 01:45:13.720564 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:13.720238 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhq7s" podUID="b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03" Sep 12 01:45:13.720564 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:13.720312 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bxnbn" podUID="f02f888c-7971-4e66-8a7d-3e180b6807d9" Sep 12 01:45:15.357245 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:15.357203 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frwnn\" (UniqueName: \"kubernetes.io/projected/f02f888c-7971-4e66-8a7d-3e180b6807d9-kube-api-access-frwnn\") pod \"network-check-target-bxnbn\" (UID: \"f02f888c-7971-4e66-8a7d-3e180b6807d9\") " pod="openshift-network-diagnostics/network-check-target-bxnbn" Sep 12 01:45:15.357811 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:15.357256 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03-metrics-certs\") pod \"network-metrics-daemon-nhq7s\" (UID: \"b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03\") " pod="openshift-multus/network-metrics-daemon-nhq7s" Sep 12 01:45:15.357811 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:15.357294 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f016f46e-b786-405e-8f6e-faca9e639b5f-original-pull-secret\") pod \"global-pull-secret-syncer-wjmgj\" (UID: \"f016f46e-b786-405e-8f6e-faca9e639b5f\") " pod="kube-system/global-pull-secret-syncer-wjmgj" Sep 12 01:45:15.357811 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:15.357403 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 12 01:45:15.357811 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:15.357411 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 12 01:45:15.357811 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:15.357404 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 12 01:45:15.357811 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:15.357496 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03-metrics-certs podName:b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03 nodeName:}" failed. No retries permitted until 2026-09-12 01:45:31.357471957 +0000 UTC m=+34.149465754 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03-metrics-certs") pod "network-metrics-daemon-nhq7s" (UID: "b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 12 01:45:15.357811 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:15.357507 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 12 01:45:15.357811 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:15.357518 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f016f46e-b786-405e-8f6e-faca9e639b5f-original-pull-secret podName:f016f46e-b786-405e-8f6e-faca9e639b5f nodeName:}" failed. No retries permitted until 2026-09-12 01:45:31.357507396 +0000 UTC m=+34.149501172 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f016f46e-b786-405e-8f6e-faca9e639b5f-original-pull-secret") pod "global-pull-secret-syncer-wjmgj" (UID: "f016f46e-b786-405e-8f6e-faca9e639b5f") : object "kube-system"/"original-pull-secret" not registered Sep 12 01:45:15.357811 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:15.357521 2579 projected.go:196] Error preparing data for projected volume kube-api-access-frwnn for pod openshift-network-diagnostics/network-check-target-bxnbn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 12 01:45:15.357811 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:15.357565 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f02f888c-7971-4e66-8a7d-3e180b6807d9-kube-api-access-frwnn podName:f02f888c-7971-4e66-8a7d-3e180b6807d9 nodeName:}" failed. No retries permitted until 2026-09-12 01:45:31.35754738 +0000 UTC m=+34.149541167 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-frwnn" (UniqueName: "kubernetes.io/projected/f02f888c-7971-4e66-8a7d-3e180b6807d9-kube-api-access-frwnn") pod "network-check-target-bxnbn" (UID: "f02f888c-7971-4e66-8a7d-3e180b6807d9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 12 01:45:15.719736 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:15.719708 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wjmgj" Sep 12 01:45:15.719736 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:15.719736 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bxnbn" Sep 12 01:45:15.720025 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:15.719819 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhq7s" Sep 12 01:45:15.720025 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:15.719831 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wjmgj" podUID="f016f46e-b786-405e-8f6e-faca9e639b5f" Sep 12 01:45:15.720025 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:15.719890 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bxnbn" podUID="f02f888c-7971-4e66-8a7d-3e180b6807d9" Sep 12 01:45:15.720025 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:15.719988 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhq7s" podUID="b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03" Sep 12 01:45:17.720690 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:17.720658 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wjmgj" Sep 12 01:45:17.721177 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:17.720770 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wjmgj" podUID="f016f46e-b786-405e-8f6e-faca9e639b5f" Sep 12 01:45:17.721177 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:17.720881 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bxnbn" Sep 12 01:45:17.721177 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:17.720973 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhq7s" Sep 12 01:45:17.721177 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:17.720994 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bxnbn" podUID="f02f888c-7971-4e66-8a7d-3e180b6807d9" Sep 12 01:45:17.721177 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:17.721086 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhq7s" podUID="b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03" Sep 12 01:45:19.719953 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:19.719916 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhq7s" Sep 12 01:45:19.719953 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:19.719959 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wjmgj" Sep 12 01:45:19.720845 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:19.719915 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bxnbn" Sep 12 01:45:19.720845 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:19.720066 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhq7s" podUID="b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03" Sep 12 01:45:19.720845 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:19.720220 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wjmgj" podUID="f016f46e-b786-405e-8f6e-faca9e639b5f" Sep 12 01:45:19.720845 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:19.720316 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bxnbn" podUID="f02f888c-7971-4e66-8a7d-3e180b6807d9" Sep 12 01:45:19.823080 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:19.823051 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4fhmh" event={"ID":"5a68d964-2017-4185-b57a-5b09bfe19c4f","Type":"ContainerStarted","Data":"4c6445166aa68c91d7dfec081bcc5d6c59861e44b99f64776ae3fe552e4bda16"} Sep 12 01:45:19.824640 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:19.824605 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qzpw5" event={"ID":"a1140d6e-61e4-4bad-a337-cc28db7dd812","Type":"ContainerStarted","Data":"0649fc6d89a5cea9b54fcefd49fd2fd0320304b5b3860c0ea77640acd9cc039a"} Sep 12 01:45:19.829255 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:19.828938 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" event={"ID":"9274dec2-bf50-42c6-8c65-7ecb88771cc7","Type":"ContainerStarted","Data":"e62cc726da74a96e0e327385e274392eb09e740258ffbfef89573d07c72bf4c4"} Sep 12 01:45:19.829255 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:19.829012 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" event={"ID":"9274dec2-bf50-42c6-8c65-7ecb88771cc7","Type":"ContainerStarted","Data":"07b0592a1544cd692974064a34edcc46d2516e20c7aa32cef49ca0033e48cb8b"} Sep 12 01:45:19.829255 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:19.829061 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" event={"ID":"9274dec2-bf50-42c6-8c65-7ecb88771cc7","Type":"ContainerStarted","Data":"8c286ddac3a24fe4e83e3e4e94fe788c36de4005cfe54c5f46a70c5e876081a1"} Sep 12 01:45:19.829255 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:19.829109 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" event={"ID":"9274dec2-bf50-42c6-8c65-7ecb88771cc7","Type":"ContainerStarted","Data":"e000fce6630861abef2d1148093815c463af13aa7326af3c8f40327e96533dfa"} Sep 12 01:45:19.829255 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:19.829155 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" event={"ID":"9274dec2-bf50-42c6-8c65-7ecb88771cc7","Type":"ContainerStarted","Data":"324588ba6e78d8557ddf64dfc7e325848bd95110ce2257c62f083b36df00c0d1"} Sep 12 01:45:19.829255 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:19.829199 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" event={"ID":"9274dec2-bf50-42c6-8c65-7ecb88771cc7","Type":"ContainerStarted","Data":"c8d9d7d0a8ba3b363faeb536f9dfa05a9fa0b76c7c225cb5851f5b59ba7361e3"} Sep 12 01:45:19.831314 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:19.830990 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2dpmw" event={"ID":"6e1eb253-a93e-4370-9cbd-a5ec2ae013a9","Type":"ContainerStarted","Data":"7d27733974fde745e498832358ad7063bae7892a9686bd468d21c9fa19cd26d9"} Sep 12 01:45:19.832906 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:19.832880 2579 generic.go:356] "Generic (PLEG): container finished" podID="46256f89-0730-441e-97d5-6ef1dff74569" containerID="ca22adb8d5d53e1d368ac2025e44b4bec28538d03962e5d427f04b77679684d0" exitCode=0 Sep 12 01:45:19.833002 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:19.832956 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rzbs6" event={"ID":"46256f89-0730-441e-97d5-6ef1dff74569","Type":"ContainerDied","Data":"ca22adb8d5d53e1d368ac2025e44b4bec28538d03962e5d427f04b77679684d0"} Sep 12 01:45:19.835167 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:19.835147 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2shh6" event={"ID":"e7d6653c-acc4-4af4-89c4-39eff31ba128","Type":"ContainerStarted","Data":"0cb7504cc202e2dd4c5a892c0b15a08f607cd313b654088089e75a2a16e48d34"} Sep 12 01:45:19.836812 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:19.836788 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-d4598" event={"ID":"e5ed1052-5fa4-4d50-be77-d799d5722140","Type":"ContainerStarted","Data":"94feaa57fd69530dcc927df9161712732ef29c478422afb60da25c13a79f4da6"} Sep 12 01:45:19.838303 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:19.838281 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nhw89" event={"ID":"b1408eab-ddcd-41d6-87af-50980dd43721","Type":"ContainerStarted","Data":"ad72294c75878395ff376e449eb1bc4bd7c2a32670394756cb30b0b9076e05d6"} Sep 12 01:45:19.839270 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:19.839221 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-79.ec2.internal" podStartSLOduration=21.839204809 podStartE2EDuration="21.839204809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 01:44:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:45:02.7953751 +0000 UTC m=+5.587368896" watchObservedRunningTime="2026-09-12 01:45:19.839204809 +0000 UTC m=+22.631198603" Sep 12 01:45:19.854828 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:19.854734 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-4fhmh" podStartSLOduration=4.714257603 podStartE2EDuration="22.854713819s" totalImagesPullingTime="18.140456216s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 01:44:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:45:19.839712239 +0000 UTC m=+22.631706033" watchObservedRunningTime="2026-09-12 01:45:19.854713819 +0000 UTC m=+22.646707616" Sep 12 01:45:19.870798 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:19.870759 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-d4598" podStartSLOduration=4.653724735 podStartE2EDuration="22.87074676s" totalImagesPullingTime="18.217022025s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 01:44:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:45:19.855430753 +0000 UTC m=+22.647424550" watchObservedRunningTime="2026-09-12 01:45:19.87074676 +0000 UTC m=+22.662740554" Sep 12 01:45:19.870988 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:19.870970 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-qzpw5" podStartSLOduration=4.762376459 podStartE2EDuration="22.870963379s" totalImagesPullingTime="18.10858692s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 01:44:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:45:19.870469207 +0000 UTC m=+22.662462998" watchObservedRunningTime="2026-09-12 01:45:19.870963379 +0000 UTC m=+22.662957174" Sep 12 01:45:19.888946 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:19.888908 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-nhw89" podStartSLOduration=12.32601121 podStartE2EDuration="22.888894293s" totalImagesPullingTime="10.562883083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:44:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:45:19.888662942 +0000 UTC m=+22.680656737" watchObservedRunningTime="2026-09-12 01:45:19.888894293 +0000 UTC m=+22.680888084" Sep 12 01:45:19.905633 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:19.905415 2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 12 01:45:19.919883 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:19.919838 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2dpmw" podStartSLOduration=8.978849607 podStartE2EDuration="22.919823014s" totalImagesPullingTime="13.940973407s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 01:44:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:45:19.919472941 +0000 UTC m=+22.711466734" watchObservedRunningTime="2026-09-12 01:45:19.919823014 +0000 UTC m=+22.711816809" Sep 12 01:45:20.688446 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:20.688353 2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-12T01:45:19.905629236Z","UUID":"5dde983b-103a-4e0a-9602-508bc2d0507f","Handler":null,"Name":"","Endpoint":""} Sep 12 01:45:20.690217 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:20.690188 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 12 01:45:20.690217 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:20.690223 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 12 01:45:20.842344 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:20.842305 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pkjc5" event={"ID":"900016e2-2d16-467b-a924-83f1aec3610e","Type":"ContainerStarted","Data":"9128849644ae3a15892a2decae3527b5aeeb4ae5b771258be7a224ebbf067e70"} Sep 12 01:45:20.845707 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:20.845598 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2shh6" event={"ID":"e7d6653c-acc4-4af4-89c4-39eff31ba128","Type":"ContainerStarted","Data":"fbc23265321edf7172439f6fcb6304e5e0a1c1eb29a80069cf831e52e7f37f40"} Sep 12 01:45:20.845707 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:20.845651 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2shh6" event={"ID":"e7d6653c-acc4-4af4-89c4-39eff31ba128","Type":"ContainerStarted","Data":"9cf8d9d8bd9f4c7cdbc4582f88874aa51d68b18b40bd35314847eb3ed9854162"} Sep 12 01:45:20.857013 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:20.856964 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-pkjc5" podStartSLOduration=5.715922421 podStartE2EDuration="23.856947549s" totalImagesPullingTime="18.141025128s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 01:44:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:45:20.856221706 +0000 UTC m=+23.648215501" watchObservedRunningTime="2026-09-12 01:45:20.856947549 +0000 UTC m=+23.648941345" Sep 12 01:45:20.871924 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:20.871862 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2shh6" podStartSLOduration=4.100763804 podStartE2EDuration="23.871842615s" totalImagesPullingTime="19.771078811s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 01:44:57 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:45:20.87134258 +0000 UTC m=+23.663336375" watchObservedRunningTime="2026-09-12 01:45:20.871842615 +0000 UTC m=+23.663836414" Sep 12 01:45:21.719925 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:21.719887 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhq7s" Sep 12 01:45:21.720105 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:21.720015 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhq7s" podUID="b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03" Sep 12 01:45:21.720105 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:21.720053 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wjmgj" Sep 12 01:45:21.720189 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:21.720123 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wjmgj" podUID="f016f46e-b786-405e-8f6e-faca9e639b5f" Sep 12 01:45:21.720189 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:21.720161 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bxnbn" Sep 12 01:45:21.720263 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:21.720202 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bxnbn" podUID="f02f888c-7971-4e66-8a7d-3e180b6807d9" Sep 12 01:45:21.851679 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:21.851595 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" event={"ID":"9274dec2-bf50-42c6-8c65-7ecb88771cc7","Type":"ContainerStarted","Data":"06207f60580745480733fb0206030324e9dc1742e3e442a5a24a972bf5f42bff"} Sep 12 01:45:21.968723 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:21.968693 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-nhw89" Sep 12 01:45:21.969293 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:21.969275 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-nhw89" Sep 12 01:45:23.720217 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:23.720176 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhq7s" Sep 12 01:45:23.720635 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:23.720236 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wjmgj" Sep 12 01:45:23.720635 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:23.720331 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bxnbn" Sep 12 01:45:23.720635 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:23.720356 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhq7s" podUID="b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03" Sep 12 01:45:23.720635 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:23.720408 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bxnbn" podUID="f02f888c-7971-4e66-8a7d-3e180b6807d9" Sep 12 01:45:23.720635 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:23.720516 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wjmgj" podUID="f016f46e-b786-405e-8f6e-faca9e639b5f" Sep 12 01:45:23.861046 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:23.861018 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" event={"ID":"9274dec2-bf50-42c6-8c65-7ecb88771cc7","Type":"ContainerStarted","Data":"c4f7e6f9f60162feb30831d2922c14842526149309164f1174de251d12ec8df8"} Sep 12 01:45:23.861287 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:23.861261 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:45:23.861377 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:23.861326 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:45:23.886877 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:23.886727 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:45:23.889899 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:23.889844 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" podStartSLOduration=8.740460299 podStartE2EDuration="26.889824458s" totalImagesPullingTime="18.149364159s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 01:44:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:45:23.889088852 +0000 UTC m=+26.681082646" watchObservedRunningTime="2026-09-12 01:45:23.889824458 +0000 UTC m=+26.681818254" Sep 12 01:45:24.865324 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:24.865290 2579 generic.go:356] "Generic (PLEG): container finished" podID="46256f89-0730-441e-97d5-6ef1dff74569" containerID="9f1bcba35c1942e6fb90c96f015d0abbe49de551abcf176e58a6f821df6b1f64" exitCode=0 Sep 12 01:45:24.865771 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:24.865380 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rzbs6" event={"ID":"46256f89-0730-441e-97d5-6ef1dff74569","Type":"ContainerDied","Data":"9f1bcba35c1942e6fb90c96f015d0abbe49de551abcf176e58a6f821df6b1f64"} Sep 12 01:45:24.866547 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:24.865999 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:45:24.881425 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:24.881404 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:45:25.504402 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:25.504379 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-nhw89" Sep 12 01:45:25.504524 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:25.504514 2579 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 12 01:45:25.505162 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:25.505134 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-nhw89" Sep 12 01:45:25.652250 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:25.652218 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wjmgj"] Sep 12 01:45:25.652424 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:25.652350 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wjmgj" Sep 12 01:45:25.652529 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:25.652504 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wjmgj" podUID="f016f46e-b786-405e-8f6e-faca9e639b5f" Sep 12 01:45:25.655278 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:25.655257 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bxnbn"] Sep 12 01:45:25.655388 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:25.655359 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bxnbn" Sep 12 01:45:25.655499 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:25.655479 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bxnbn" podUID="f02f888c-7971-4e66-8a7d-3e180b6807d9" Sep 12 01:45:25.655938 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:25.655919 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nhq7s"] Sep 12 01:45:25.656034 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:25.656019 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhq7s" Sep 12 01:45:25.656156 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:25.656134 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhq7s" podUID="b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03" Sep 12 01:45:26.872414 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:26.872199 2579 generic.go:356] "Generic (PLEG): container finished" podID="46256f89-0730-441e-97d5-6ef1dff74569" containerID="356c7c9c55e2313117b7b17c9cf26373e7696cc1a00123640fc844b049954b8e" exitCode=0 Sep 12 01:45:26.873233 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:26.872280 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rzbs6" event={"ID":"46256f89-0730-441e-97d5-6ef1dff74569","Type":"ContainerDied","Data":"356c7c9c55e2313117b7b17c9cf26373e7696cc1a00123640fc844b049954b8e"} Sep 12 01:45:27.721324 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:27.721254 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wjmgj" Sep 12 01:45:27.721500 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:27.721345 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wjmgj" podUID="f016f46e-b786-405e-8f6e-faca9e639b5f" Sep 12 01:45:27.721500 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:27.721431 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhq7s" Sep 12 01:45:27.721587 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:27.721557 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhq7s" podUID="b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03" Sep 12 01:45:27.721621 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:27.721613 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bxnbn" Sep 12 01:45:27.721696 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:27.721679 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bxnbn" podUID="f02f888c-7971-4e66-8a7d-3e180b6807d9" Sep 12 01:45:27.877670 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:27.877553 2579 generic.go:356] "Generic (PLEG): container finished" podID="46256f89-0730-441e-97d5-6ef1dff74569" containerID="3a0ed9761a06b3b428c6c1d92b7d4f8e44a3ef15bbdad7163538cf03aaa96c8a" exitCode=0 Sep 12 01:45:27.877670 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:27.877615 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rzbs6" event={"ID":"46256f89-0730-441e-97d5-6ef1dff74569","Type":"ContainerDied","Data":"3a0ed9761a06b3b428c6c1d92b7d4f8e44a3ef15bbdad7163538cf03aaa96c8a"} Sep 12 01:45:27.994258 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:27.994182 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:28.094540 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:28.094509 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:28.934783 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:28.934742 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:29.034664 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:29.034614 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:29.136288 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:29.136256 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:29.192920 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:29.192830 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:29.720731 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:29.720695 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bxnbn" Sep 12 01:45:29.720913 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:29.720695 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wjmgj" Sep 12 01:45:29.720913 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:29.720808 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bxnbn" podUID="f02f888c-7971-4e66-8a7d-3e180b6807d9" Sep 12 01:45:29.721034 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:29.720906 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wjmgj" podUID="f016f46e-b786-405e-8f6e-faca9e639b5f" Sep 12 01:45:29.721034 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:29.720695 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhq7s" Sep 12 01:45:29.721101 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:29.721035 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhq7s" podUID="b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03" Sep 12 01:45:29.889004 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:29.888969 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:31.378664 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.378631 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frwnn\" (UniqueName: \"kubernetes.io/projected/f02f888c-7971-4e66-8a7d-3e180b6807d9-kube-api-access-frwnn\") pod \"network-check-target-bxnbn\" (UID: \"f02f888c-7971-4e66-8a7d-3e180b6807d9\") " pod="openshift-network-diagnostics/network-check-target-bxnbn" Sep 12 01:45:31.379094 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.378674 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03-metrics-certs\") pod \"network-metrics-daemon-nhq7s\" (UID: \"b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03\") " pod="openshift-multus/network-metrics-daemon-nhq7s" Sep 12 01:45:31.379094 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.378707 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f016f46e-b786-405e-8f6e-faca9e639b5f-original-pull-secret\") pod \"global-pull-secret-syncer-wjmgj\" (UID: \"f016f46e-b786-405e-8f6e-faca9e639b5f\") " pod="kube-system/global-pull-secret-syncer-wjmgj" Sep 12 01:45:31.379094 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:31.378800 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 12 01:45:31.379094 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:31.378813 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 12 01:45:31.379094 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:31.378832 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 12 01:45:31.379094 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:31.378843 2579 projected.go:196] Error preparing data for projected volume kube-api-access-frwnn for pod openshift-network-diagnostics/network-check-target-bxnbn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 12 01:45:31.379094 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:31.378870 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03-metrics-certs podName:b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03 nodeName:}" failed. No retries permitted until 2026-09-12 01:46:03.378849841 +0000 UTC m=+66.170843615 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03-metrics-certs") pod "network-metrics-daemon-nhq7s" (UID: "b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 12 01:45:31.379094 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:31.378806 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 12 01:45:31.379094 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:31.378888 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f02f888c-7971-4e66-8a7d-3e180b6807d9-kube-api-access-frwnn podName:f02f888c-7971-4e66-8a7d-3e180b6807d9 nodeName:}" failed. No retries permitted until 2026-09-12 01:46:03.378878684 +0000 UTC m=+66.170872457 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-frwnn" (UniqueName: "kubernetes.io/projected/f02f888c-7971-4e66-8a7d-3e180b6807d9-kube-api-access-frwnn") pod "network-check-target-bxnbn" (UID: "f02f888c-7971-4e66-8a7d-3e180b6807d9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 12 01:45:31.379094 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:31.378905 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f016f46e-b786-405e-8f6e-faca9e639b5f-original-pull-secret podName:f016f46e-b786-405e-8f6e-faca9e639b5f nodeName:}" failed. No retries permitted until 2026-09-12 01:46:03.378894595 +0000 UTC m=+66.170888382 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f016f46e-b786-405e-8f6e-faca9e639b5f-original-pull-secret") pod "global-pull-secret-syncer-wjmgj" (UID: "f016f46e-b786-405e-8f6e-faca9e639b5f") : object "kube-system"/"original-pull-secret" not registered Sep 12 01:45:31.472252 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.472227 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-79.ec2.internal" event="NodeReady" Sep 12 01:45:31.472429 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.472380 2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 12 01:45:31.504649 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.504572 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-cbdcf5d7-xg922"] Sep 12 01:45:31.519312 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.519281 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-p96nw"] Sep 12 01:45:31.520219 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.520194 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cbdcf5d7-xg922" Sep 12 01:45:31.522876 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.522853 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 12 01:45:31.523147 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.523127 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-2s9js\"" Sep 12 01:45:31.523773 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.523753 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 12 01:45:31.523879 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.523767 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 12 01:45:31.533522 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.531943 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-p96nw" Sep 12 01:45:31.533522 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.531959 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 12 01:45:31.533522 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.531977 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8bm2n"] Sep 12 01:45:31.534417 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.534398 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 12 01:45:31.534831 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.534813 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 12 01:45:31.535368 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.535003 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gmqgn\"" Sep 12 01:45:31.542268 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.542249 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-cbdcf5d7-xg922"] Sep 12 01:45:31.542370 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.542285 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8bm2n"] Sep 12 01:45:31.542370 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.542339 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-p96nw"] Sep 12 01:45:31.542489 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.542427 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8bm2n" Sep 12 01:45:31.544997 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.544982 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-5l94s\"" Sep 12 01:45:31.545322 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.545302 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 12 01:45:31.545474 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.545442 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 12 01:45:31.545613 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.545284 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 12 01:45:31.680980 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.680945 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-image-registry-private-configuration\") pod \"image-registry-cbdcf5d7-xg922\" (UID: \"5e0ab63f-37d5-4aa5-aeca-4575b5063d13\") " pod="openshift-image-registry/image-registry-cbdcf5d7-xg922" Sep 12 01:45:31.680980 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.680987 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2d765cc-6edd-4ba0-b8b0-81391dd5aa3d-tmp-dir\") pod \"dns-default-p96nw\" (UID: \"f2d765cc-6edd-4ba0-b8b0-81391dd5aa3d\") " pod="openshift-dns/dns-default-p96nw" Sep 12 01:45:31.681265 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.681012 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzbzj\" (UniqueName: \"kubernetes.io/projected/f2d765cc-6edd-4ba0-b8b0-81391dd5aa3d-kube-api-access-kzbzj\") pod \"dns-default-p96nw\" (UID: \"f2d765cc-6edd-4ba0-b8b0-81391dd5aa3d\") " pod="openshift-dns/dns-default-p96nw" Sep 12 01:45:31.681265 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.681137 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-registry-tls\") pod \"image-registry-cbdcf5d7-xg922\" (UID: \"5e0ab63f-37d5-4aa5-aeca-4575b5063d13\") " pod="openshift-image-registry/image-registry-cbdcf5d7-xg922" Sep 12 01:45:31.681265 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.681164 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-trusted-ca\") pod \"image-registry-cbdcf5d7-xg922\" (UID: \"5e0ab63f-37d5-4aa5-aeca-4575b5063d13\") " pod="openshift-image-registry/image-registry-cbdcf5d7-xg922" Sep 12 01:45:31.681265 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.681183 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f844\" (UniqueName: \"kubernetes.io/projected/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-kube-api-access-9f844\") pod \"image-registry-cbdcf5d7-xg922\" (UID: \"5e0ab63f-37d5-4aa5-aeca-4575b5063d13\") " pod="openshift-image-registry/image-registry-cbdcf5d7-xg922" Sep 12 01:45:31.681265 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.681219 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-installation-pull-secrets\") pod \"image-registry-cbdcf5d7-xg922\" (UID: \"5e0ab63f-37d5-4aa5-aeca-4575b5063d13\") " pod="openshift-image-registry/image-registry-cbdcf5d7-xg922" Sep 12 01:45:31.681265 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.681241 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-bound-sa-token\") pod \"image-registry-cbdcf5d7-xg922\" (UID: \"5e0ab63f-37d5-4aa5-aeca-4575b5063d13\") " pod="openshift-image-registry/image-registry-cbdcf5d7-xg922" Sep 12 01:45:31.681265 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.681259 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c84a24ed-159c-4497-863f-03a5a9c0d81d-cert\") pod \"ingress-canary-8bm2n\" (UID: \"c84a24ed-159c-4497-863f-03a5a9c0d81d\") " pod="openshift-ingress-canary/ingress-canary-8bm2n" Sep 12 01:45:31.681605 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.681277 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtkqm\" (UniqueName: \"kubernetes.io/projected/c84a24ed-159c-4497-863f-03a5a9c0d81d-kube-api-access-gtkqm\") pod \"ingress-canary-8bm2n\" (UID: \"c84a24ed-159c-4497-863f-03a5a9c0d81d\") " pod="openshift-ingress-canary/ingress-canary-8bm2n" Sep 12 01:45:31.681605 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.681302 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-registry-certificates\") pod \"image-registry-cbdcf5d7-xg922\" (UID: \"5e0ab63f-37d5-4aa5-aeca-4575b5063d13\") " pod="openshift-image-registry/image-registry-cbdcf5d7-xg922" Sep 12 01:45:31.681605 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.681371 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-ca-trust-extracted\") pod \"image-registry-cbdcf5d7-xg922\" (UID: \"5e0ab63f-37d5-4aa5-aeca-4575b5063d13\") " pod="openshift-image-registry/image-registry-cbdcf5d7-xg922" Sep 12 01:45:31.681605 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.681404 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f2d765cc-6edd-4ba0-b8b0-81391dd5aa3d-config-volume\") pod \"dns-default-p96nw\" (UID: \"f2d765cc-6edd-4ba0-b8b0-81391dd5aa3d\") " pod="openshift-dns/dns-default-p96nw" Sep 12 01:45:31.681605 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.681437 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2d765cc-6edd-4ba0-b8b0-81391dd5aa3d-metrics-tls\") pod \"dns-default-p96nw\" (UID: \"f2d765cc-6edd-4ba0-b8b0-81391dd5aa3d\") " pod="openshift-dns/dns-default-p96nw" Sep 12 01:45:31.720415 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.720382 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bxnbn" Sep 12 01:45:31.720594 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.720382 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wjmgj" Sep 12 01:45:31.720594 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.720512 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhq7s" Sep 12 01:45:31.723096 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.723076 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 12 01:45:31.723096 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.723086 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hmzff\"" Sep 12 01:45:31.723287 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.723136 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fqtlc\"" Sep 12 01:45:31.723287 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.723175 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 12 01:45:31.723392 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.723315 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 12 01:45:31.723392 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.723356 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 12 01:45:31.782768 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.782699 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-registry-certificates\") pod \"image-registry-cbdcf5d7-xg922\" (UID: \"5e0ab63f-37d5-4aa5-aeca-4575b5063d13\") " pod="openshift-image-registry/image-registry-cbdcf5d7-xg922" Sep 12 01:45:31.782768 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.782734 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-ca-trust-extracted\") pod \"image-registry-cbdcf5d7-xg922\" (UID: \"5e0ab63f-37d5-4aa5-aeca-4575b5063d13\") " pod="openshift-image-registry/image-registry-cbdcf5d7-xg922" Sep 12 01:45:31.782768 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.782753 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f2d765cc-6edd-4ba0-b8b0-81391dd5aa3d-config-volume\") pod \"dns-default-p96nw\" (UID: \"f2d765cc-6edd-4ba0-b8b0-81391dd5aa3d\") " pod="openshift-dns/dns-default-p96nw" Sep 12 01:45:31.782983 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.782852 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2d765cc-6edd-4ba0-b8b0-81391dd5aa3d-metrics-tls\") pod \"dns-default-p96nw\" (UID: \"f2d765cc-6edd-4ba0-b8b0-81391dd5aa3d\") " pod="openshift-dns/dns-default-p96nw" Sep 12 01:45:31.782983 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.782898 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-image-registry-private-configuration\") pod \"image-registry-cbdcf5d7-xg922\" (UID: \"5e0ab63f-37d5-4aa5-aeca-4575b5063d13\") " pod="openshift-image-registry/image-registry-cbdcf5d7-xg922" Sep 12 01:45:31.783051 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:31.783009 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 12 01:45:31.783097 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.783058 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-ca-trust-extracted\") pod \"image-registry-cbdcf5d7-xg922\" (UID: \"5e0ab63f-37d5-4aa5-aeca-4575b5063d13\") " pod="openshift-image-registry/image-registry-cbdcf5d7-xg922" Sep 12 01:45:31.783097 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:31.783086 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2d765cc-6edd-4ba0-b8b0-81391dd5aa3d-metrics-tls podName:f2d765cc-6edd-4ba0-b8b0-81391dd5aa3d nodeName:}" failed. No retries permitted until 2026-09-12 01:45:32.283063507 +0000 UTC m=+35.075057301 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f2d765cc-6edd-4ba0-b8b0-81391dd5aa3d-metrics-tls") pod "dns-default-p96nw" (UID: "f2d765cc-6edd-4ba0-b8b0-81391dd5aa3d") : secret "dns-default-metrics-tls" not found Sep 12 01:45:31.783182 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.783111 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2d765cc-6edd-4ba0-b8b0-81391dd5aa3d-tmp-dir\") pod \"dns-default-p96nw\" (UID: \"f2d765cc-6edd-4ba0-b8b0-81391dd5aa3d\") " pod="openshift-dns/dns-default-p96nw" Sep 12 01:45:31.783182 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.783143 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzbzj\" (UniqueName: \"kubernetes.io/projected/f2d765cc-6edd-4ba0-b8b0-81391dd5aa3d-kube-api-access-kzbzj\") pod \"dns-default-p96nw\" (UID: \"f2d765cc-6edd-4ba0-b8b0-81391dd5aa3d\") " pod="openshift-dns/dns-default-p96nw" Sep 12 01:45:31.783265 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.783235 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-registry-tls\") pod \"image-registry-cbdcf5d7-xg922\" (UID: \"5e0ab63f-37d5-4aa5-aeca-4575b5063d13\") " pod="openshift-image-registry/image-registry-cbdcf5d7-xg922" Sep 12 01:45:31.783319 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.783264 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-trusted-ca\") pod \"image-registry-cbdcf5d7-xg922\" (UID: \"5e0ab63f-37d5-4aa5-aeca-4575b5063d13\") " pod="openshift-image-registry/image-registry-cbdcf5d7-xg922" Sep 12 01:45:31.783368 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.783337 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-registry-certificates\") pod \"image-registry-cbdcf5d7-xg922\" (UID: \"5e0ab63f-37d5-4aa5-aeca-4575b5063d13\") " pod="openshift-image-registry/image-registry-cbdcf5d7-xg922" Sep 12 01:45:31.783420 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.783346 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2d765cc-6edd-4ba0-b8b0-81391dd5aa3d-tmp-dir\") pod \"dns-default-p96nw\" (UID: \"f2d765cc-6edd-4ba0-b8b0-81391dd5aa3d\") " pod="openshift-dns/dns-default-p96nw" Sep 12 01:45:31.783420 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.783366 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f2d765cc-6edd-4ba0-b8b0-81391dd5aa3d-config-volume\") pod \"dns-default-p96nw\" (UID: \"f2d765cc-6edd-4ba0-b8b0-81391dd5aa3d\") " pod="openshift-dns/dns-default-p96nw" Sep 12 01:45:31.783542 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:31.783434 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 12 01:45:31.783542 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:31.783468 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cbdcf5d7-xg922: secret "image-registry-tls" not found Sep 12 01:45:31.783542 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.783484 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f844\" (UniqueName: \"kubernetes.io/projected/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-kube-api-access-9f844\") pod \"image-registry-cbdcf5d7-xg922\" (UID: \"5e0ab63f-37d5-4aa5-aeca-4575b5063d13\") " pod="openshift-image-registry/image-registry-cbdcf5d7-xg922" Sep 12 01:45:31.783542 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:31.783520 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-registry-tls podName:5e0ab63f-37d5-4aa5-aeca-4575b5063d13 nodeName:}" failed. No retries permitted until 2026-09-12 01:45:32.283505157 +0000 UTC m=+35.075498929 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-registry-tls") pod "image-registry-cbdcf5d7-xg922" (UID: "5e0ab63f-37d5-4aa5-aeca-4575b5063d13") : secret "image-registry-tls" not found Sep 12 01:45:31.783740 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.783544 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-installation-pull-secrets\") pod \"image-registry-cbdcf5d7-xg922\" (UID: \"5e0ab63f-37d5-4aa5-aeca-4575b5063d13\") " pod="openshift-image-registry/image-registry-cbdcf5d7-xg922" Sep 12 01:45:31.783740 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.783566 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-bound-sa-token\") pod \"image-registry-cbdcf5d7-xg922\" (UID: \"5e0ab63f-37d5-4aa5-aeca-4575b5063d13\") " pod="openshift-image-registry/image-registry-cbdcf5d7-xg922" Sep 12 01:45:31.783740 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.783587 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c84a24ed-159c-4497-863f-03a5a9c0d81d-cert\") pod \"ingress-canary-8bm2n\" (UID: \"c84a24ed-159c-4497-863f-03a5a9c0d81d\") " pod="openshift-ingress-canary/ingress-canary-8bm2n" Sep 12 01:45:31.783740 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.783614 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtkqm\" (UniqueName: \"kubernetes.io/projected/c84a24ed-159c-4497-863f-03a5a9c0d81d-kube-api-access-gtkqm\") pod \"ingress-canary-8bm2n\" (UID: \"c84a24ed-159c-4497-863f-03a5a9c0d81d\") " pod="openshift-ingress-canary/ingress-canary-8bm2n" Sep 12 01:45:31.784050 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.784028 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-trusted-ca\") pod \"image-registry-cbdcf5d7-xg922\" (UID: \"5e0ab63f-37d5-4aa5-aeca-4575b5063d13\") " pod="openshift-image-registry/image-registry-cbdcf5d7-xg922" Sep 12 01:45:31.787332 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.787314 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-image-registry-private-configuration\") pod \"image-registry-cbdcf5d7-xg922\" (UID: \"5e0ab63f-37d5-4aa5-aeca-4575b5063d13\") " pod="openshift-image-registry/image-registry-cbdcf5d7-xg922" Sep 12 01:45:31.787421 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.787343 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-installation-pull-secrets\") pod \"image-registry-cbdcf5d7-xg922\" (UID: \"5e0ab63f-37d5-4aa5-aeca-4575b5063d13\") " pod="openshift-image-registry/image-registry-cbdcf5d7-xg922" Sep 12 01:45:31.787421 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.787412 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c84a24ed-159c-4497-863f-03a5a9c0d81d-cert\") pod \"ingress-canary-8bm2n\" (UID: \"c84a24ed-159c-4497-863f-03a5a9c0d81d\") " pod="openshift-ingress-canary/ingress-canary-8bm2n" Sep 12 01:45:31.791873 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.791851 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzbzj\" (UniqueName: \"kubernetes.io/projected/f2d765cc-6edd-4ba0-b8b0-81391dd5aa3d-kube-api-access-kzbzj\") pod \"dns-default-p96nw\" (UID: \"f2d765cc-6edd-4ba0-b8b0-81391dd5aa3d\") " pod="openshift-dns/dns-default-p96nw" Sep 12 01:45:31.792134 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.792116 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f844\" (UniqueName: \"kubernetes.io/projected/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-kube-api-access-9f844\") pod \"image-registry-cbdcf5d7-xg922\" (UID: \"5e0ab63f-37d5-4aa5-aeca-4575b5063d13\") " pod="openshift-image-registry/image-registry-cbdcf5d7-xg922" Sep 12 01:45:31.792241 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.792219 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtkqm\" (UniqueName: \"kubernetes.io/projected/c84a24ed-159c-4497-863f-03a5a9c0d81d-kube-api-access-gtkqm\") pod \"ingress-canary-8bm2n\" (UID: \"c84a24ed-159c-4497-863f-03a5a9c0d81d\") " pod="openshift-ingress-canary/ingress-canary-8bm2n" Sep 12 01:45:31.792329 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.792315 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-bound-sa-token\") pod \"image-registry-cbdcf5d7-xg922\" (UID: \"5e0ab63f-37d5-4aa5-aeca-4575b5063d13\") " pod="openshift-image-registry/image-registry-cbdcf5d7-xg922" Sep 12 01:45:31.853618 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:31.853587 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8bm2n" Sep 12 01:45:32.146470 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:32.146420 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8bm2n"] Sep 12 01:45:32.149500 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:45:32.149477 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc84a24ed_159c_4497_863f_03a5a9c0d81d.slice/crio-bdc80a3a2adcbfdae5c75ab7ec7880fe5d7c282e1e7902d45602cfb3442b37b9 WatchSource:0}: Error finding container bdc80a3a2adcbfdae5c75ab7ec7880fe5d7c282e1e7902d45602cfb3442b37b9: Status 404 returned error can't find the container with id bdc80a3a2adcbfdae5c75ab7ec7880fe5d7c282e1e7902d45602cfb3442b37b9 Sep 12 01:45:32.286488 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:32.286446 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-registry-tls\") pod \"image-registry-cbdcf5d7-xg922\" (UID: \"5e0ab63f-37d5-4aa5-aeca-4575b5063d13\") " pod="openshift-image-registry/image-registry-cbdcf5d7-xg922" Sep 12 01:45:32.286613 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:32.286514 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2d765cc-6edd-4ba0-b8b0-81391dd5aa3d-metrics-tls\") pod \"dns-default-p96nw\" (UID: \"f2d765cc-6edd-4ba0-b8b0-81391dd5aa3d\") " pod="openshift-dns/dns-default-p96nw" Sep 12 01:45:32.286613 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:32.286569 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 12 01:45:32.286613 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:32.286591 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cbdcf5d7-xg922: secret "image-registry-tls" not found Sep 12 01:45:32.286613 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:32.286595 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 12 01:45:32.286737 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:32.286642 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2d765cc-6edd-4ba0-b8b0-81391dd5aa3d-metrics-tls podName:f2d765cc-6edd-4ba0-b8b0-81391dd5aa3d nodeName:}" failed. No retries permitted until 2026-09-12 01:45:33.286629963 +0000 UTC m=+36.078623736 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f2d765cc-6edd-4ba0-b8b0-81391dd5aa3d-metrics-tls") pod "dns-default-p96nw" (UID: "f2d765cc-6edd-4ba0-b8b0-81391dd5aa3d") : secret "dns-default-metrics-tls" not found Sep 12 01:45:32.286737 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:32.286678 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-registry-tls podName:5e0ab63f-37d5-4aa5-aeca-4575b5063d13 nodeName:}" failed. No retries permitted until 2026-09-12 01:45:33.286663486 +0000 UTC m=+36.078657259 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-registry-tls") pod "image-registry-cbdcf5d7-xg922" (UID: "5e0ab63f-37d5-4aa5-aeca-4575b5063d13") : secret "image-registry-tls" not found Sep 12 01:45:32.890219 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:32.890175 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8bm2n" event={"ID":"c84a24ed-159c-4497-863f-03a5a9c0d81d","Type":"ContainerStarted","Data":"bdc80a3a2adcbfdae5c75ab7ec7880fe5d7c282e1e7902d45602cfb3442b37b9"} Sep 12 01:45:32.894121 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:32.894087 2579 generic.go:356] "Generic (PLEG): container finished" podID="46256f89-0730-441e-97d5-6ef1dff74569" containerID="4ee14fb3fdef781e28457356b886fea895fabce0b7503893bfc7f07c5a1ac2c0" exitCode=0 Sep 12 01:45:32.894262 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:32.894129 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rzbs6" event={"ID":"46256f89-0730-441e-97d5-6ef1dff74569","Type":"ContainerDied","Data":"4ee14fb3fdef781e28457356b886fea895fabce0b7503893bfc7f07c5a1ac2c0"} Sep 12 01:45:33.292944 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:33.292910 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:33.294759 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:33.294736 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2d765cc-6edd-4ba0-b8b0-81391dd5aa3d-metrics-tls\") pod \"dns-default-p96nw\" (UID: \"f2d765cc-6edd-4ba0-b8b0-81391dd5aa3d\") " pod="openshift-dns/dns-default-p96nw" Sep 12 01:45:33.294909 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:33.294805 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-registry-tls\") pod \"image-registry-cbdcf5d7-xg922\" (UID: \"5e0ab63f-37d5-4aa5-aeca-4575b5063d13\") " pod="openshift-image-registry/image-registry-cbdcf5d7-xg922" Sep 12 01:45:33.294909 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:33.294900 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 12 01:45:33.295015 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:33.294904 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 12 01:45:33.295015 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:33.294926 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cbdcf5d7-xg922: secret "image-registry-tls" not found Sep 12 01:45:33.295015 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:33.294971 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2d765cc-6edd-4ba0-b8b0-81391dd5aa3d-metrics-tls podName:f2d765cc-6edd-4ba0-b8b0-81391dd5aa3d nodeName:}" failed. No retries permitted until 2026-09-12 01:45:35.294952582 +0000 UTC m=+38.086946359 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f2d765cc-6edd-4ba0-b8b0-81391dd5aa3d-metrics-tls") pod "dns-default-p96nw" (UID: "f2d765cc-6edd-4ba0-b8b0-81391dd5aa3d") : secret "dns-default-metrics-tls" not found Sep 12 01:45:33.295015 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:33.294994 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-registry-tls podName:5e0ab63f-37d5-4aa5-aeca-4575b5063d13 nodeName:}" failed. No retries permitted until 2026-09-12 01:45:35.29498387 +0000 UTC m=+38.086977647 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-registry-tls") pod "image-registry-cbdcf5d7-xg922" (UID: "5e0ab63f-37d5-4aa5-aeca-4575b5063d13") : secret "image-registry-tls" not found Sep 12 01:45:33.390186 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:33.390152 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:33.900324 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:33.900285 2579 generic.go:356] "Generic (PLEG): container finished" podID="46256f89-0730-441e-97d5-6ef1dff74569" containerID="d23e037e7574210ee30a0b7b3c259641f57e23b0023af876ed65eeba6c3ddac3" exitCode=0 Sep 12 01:45:33.900854 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:33.900344 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rzbs6" event={"ID":"46256f89-0730-441e-97d5-6ef1dff74569","Type":"ContainerDied","Data":"d23e037e7574210ee30a0b7b3c259641f57e23b0023af876ed65eeba6c3ddac3"} Sep 12 01:45:34.382193 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:34.382152 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:34.483880 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:34.483851 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:34.904321 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:34.904041 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8bm2n" event={"ID":"c84a24ed-159c-4497-863f-03a5a9c0d81d","Type":"ContainerStarted","Data":"03f4ee16b5a03883c276c9aa7f1a252430778ccd5c256328c65693fbd566ae48"} Sep 12 01:45:34.907681 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:34.907658 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rzbs6" event={"ID":"46256f89-0730-441e-97d5-6ef1dff74569","Type":"ContainerStarted","Data":"8e2ce52b8447ecce22e60db014bb362564dabeb3311578c23e3ee66b6bbec7d1"} Sep 12 01:45:34.923380 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:34.923333 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8bm2n" podStartSLOduration=1.533079575 podStartE2EDuration="3.923314639s" totalImagesPullingTime="2.390235064s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:45:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:45:34.922696751 +0000 UTC m=+37.714690543" watchObservedRunningTime="2026-09-12 01:45:34.923314639 +0000 UTC m=+37.715308436" Sep 12 01:45:34.943838 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:34.943781 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-rzbs6" podStartSLOduration=10.147016685 podStartE2EDuration="37.943759275s" totalImagesPullingTime="27.79674259s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 01:44:57 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:45:34.942018036 +0000 UTC m=+37.734011831" watchObservedRunningTime="2026-09-12 01:45:34.943759275 +0000 UTC m=+37.735753072" Sep 12 01:45:35.005089 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:35.005057 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-nnp46"] Sep 12 01:45:35.023016 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:35.022993 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-nnp46"] Sep 12 01:45:35.023148 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:35.023081 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nnp46" Sep 12 01:45:35.025416 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:35.025386 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 12 01:45:35.025588 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:35.025574 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 12 01:45:35.025642 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:35.025618 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-4l6hw\"" Sep 12 01:45:35.044067 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:35.044037 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:35.106260 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:35.106229 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:35.109229 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:35.109211 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/af0906c4-27c2-4ff2-8c04-7f121cc59a3a-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-nnp46\" (UID: \"af0906c4-27c2-4ff2-8c04-7f121cc59a3a\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nnp46" Sep 12 01:45:35.109301 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:35.109260 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/af0906c4-27c2-4ff2-8c04-7f121cc59a3a-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-nnp46\" (UID: \"af0906c4-27c2-4ff2-8c04-7f121cc59a3a\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nnp46" Sep 12 01:45:35.209792 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:35.209759 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/af0906c4-27c2-4ff2-8c04-7f121cc59a3a-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-nnp46\" (UID: \"af0906c4-27c2-4ff2-8c04-7f121cc59a3a\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nnp46" Sep 12 01:45:35.209916 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:35.209816 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/af0906c4-27c2-4ff2-8c04-7f121cc59a3a-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-nnp46\" (UID: \"af0906c4-27c2-4ff2-8c04-7f121cc59a3a\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nnp46" Sep 12 01:45:35.209967 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:35.209907 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 12 01:45:35.210010 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:35.209972 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af0906c4-27c2-4ff2-8c04-7f121cc59a3a-networking-console-plugin-cert podName:af0906c4-27c2-4ff2-8c04-7f121cc59a3a nodeName:}" failed. No retries permitted until 2026-09-12 01:45:35.709956773 +0000 UTC m=+38.501950545 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/af0906c4-27c2-4ff2-8c04-7f121cc59a3a-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-nnp46" (UID: "af0906c4-27c2-4ff2-8c04-7f121cc59a3a") : secret "networking-console-plugin-cert" not found Sep 12 01:45:35.210388 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:35.210371 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/af0906c4-27c2-4ff2-8c04-7f121cc59a3a-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-nnp46\" (UID: \"af0906c4-27c2-4ff2-8c04-7f121cc59a3a\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nnp46" Sep 12 01:45:35.310381 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:35.310350 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-registry-tls\") pod \"image-registry-cbdcf5d7-xg922\" (UID: \"5e0ab63f-37d5-4aa5-aeca-4575b5063d13\") " pod="openshift-image-registry/image-registry-cbdcf5d7-xg922" Sep 12 01:45:35.310555 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:35.310402 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2d765cc-6edd-4ba0-b8b0-81391dd5aa3d-metrics-tls\") pod \"dns-default-p96nw\" (UID: \"f2d765cc-6edd-4ba0-b8b0-81391dd5aa3d\") " pod="openshift-dns/dns-default-p96nw" Sep 12 01:45:35.310555 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:35.310533 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 12 01:45:35.310633 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:35.310584 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2d765cc-6edd-4ba0-b8b0-81391dd5aa3d-metrics-tls podName:f2d765cc-6edd-4ba0-b8b0-81391dd5aa3d nodeName:}" failed. No retries permitted until 2026-09-12 01:45:39.310569987 +0000 UTC m=+42.102563770 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f2d765cc-6edd-4ba0-b8b0-81391dd5aa3d-metrics-tls") pod "dns-default-p96nw" (UID: "f2d765cc-6edd-4ba0-b8b0-81391dd5aa3d") : secret "dns-default-metrics-tls" not found Sep 12 01:45:35.310633 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:35.310590 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 12 01:45:35.310633 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:35.310607 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cbdcf5d7-xg922: secret "image-registry-tls" not found Sep 12 01:45:35.310741 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:35.310651 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-registry-tls podName:5e0ab63f-37d5-4aa5-aeca-4575b5063d13 nodeName:}" failed. No retries permitted until 2026-09-12 01:45:39.310638586 +0000 UTC m=+42.102632359 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-registry-tls") pod "image-registry-cbdcf5d7-xg922" (UID: "5e0ab63f-37d5-4aa5-aeca-4575b5063d13") : secret "image-registry-tls" not found Sep 12 01:45:35.714185 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:35.714149 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/af0906c4-27c2-4ff2-8c04-7f121cc59a3a-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-nnp46\" (UID: \"af0906c4-27c2-4ff2-8c04-7f121cc59a3a\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nnp46" Sep 12 01:45:35.714337 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:35.714254 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 12 01:45:35.714337 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:35.714315 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af0906c4-27c2-4ff2-8c04-7f121cc59a3a-networking-console-plugin-cert podName:af0906c4-27c2-4ff2-8c04-7f121cc59a3a nodeName:}" failed. No retries permitted until 2026-09-12 01:45:36.714301354 +0000 UTC m=+39.506295127 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/af0906c4-27c2-4ff2-8c04-7f121cc59a3a-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-nnp46" (UID: "af0906c4-27c2-4ff2-8c04-7f121cc59a3a") : secret "networking-console-plugin-cert" not found Sep 12 01:45:36.722300 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:36.722259 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/af0906c4-27c2-4ff2-8c04-7f121cc59a3a-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-nnp46\" (UID: \"af0906c4-27c2-4ff2-8c04-7f121cc59a3a\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nnp46" Sep 12 01:45:36.722692 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:36.722392 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 12 01:45:36.722692 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:36.722446 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af0906c4-27c2-4ff2-8c04-7f121cc59a3a-networking-console-plugin-cert podName:af0906c4-27c2-4ff2-8c04-7f121cc59a3a nodeName:}" failed. No retries permitted until 2026-09-12 01:45:38.722432527 +0000 UTC m=+41.514426300 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/af0906c4-27c2-4ff2-8c04-7f121cc59a3a-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-nnp46" (UID: "af0906c4-27c2-4ff2-8c04-7f121cc59a3a") : secret "networking-console-plugin-cert" not found Sep 12 01:45:36.835092 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:36.835057 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:37.554725 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:37.554686 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:37.654633 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:37.654600 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:38.556407 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:38.556369 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:38.735793 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:38.735754 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/af0906c4-27c2-4ff2-8c04-7f121cc59a3a-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-nnp46\" (UID: \"af0906c4-27c2-4ff2-8c04-7f121cc59a3a\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nnp46" Sep 12 01:45:38.735928 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:38.735909 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 12 01:45:38.735983 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:38.735973 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af0906c4-27c2-4ff2-8c04-7f121cc59a3a-networking-console-plugin-cert podName:af0906c4-27c2-4ff2-8c04-7f121cc59a3a nodeName:}" failed. No retries permitted until 2026-09-12 01:45:42.735959742 +0000 UTC m=+45.527953515 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/af0906c4-27c2-4ff2-8c04-7f121cc59a3a-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-nnp46" (UID: "af0906c4-27c2-4ff2-8c04-7f121cc59a3a") : secret "networking-console-plugin-cert" not found Sep 12 01:45:38.778808 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:38.778773 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-gdkkk"] Sep 12 01:45:38.783975 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:38.783947 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:38.811243 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:38.811178 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gdkkk"] Sep 12 01:45:38.811351 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:38.811285 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gdkkk" Sep 12 01:45:38.813968 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:38.813945 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 12 01:45:38.814096 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:38.813982 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 12 01:45:38.814096 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:38.813997 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 12 01:45:38.814096 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:38.814022 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9fddj\"" Sep 12 01:45:38.814902 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:38.814888 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 12 01:45:38.844313 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:38.844283 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:38.937499 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:38.937471 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7d6c62d0-83e1-41e7-8780-5dc9be2074a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gdkkk\" (UID: \"7d6c62d0-83e1-41e7-8780-5dc9be2074a5\") " pod="openshift-insights/insights-runtime-extractor-gdkkk" Sep 12 01:45:38.937625 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:38.937529 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7d6c62d0-83e1-41e7-8780-5dc9be2074a5-crio-socket\") pod \"insights-runtime-extractor-gdkkk\" (UID: \"7d6c62d0-83e1-41e7-8780-5dc9be2074a5\") " pod="openshift-insights/insights-runtime-extractor-gdkkk" Sep 12 01:45:38.937625 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:38.937555 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrqmb\" (UniqueName: \"kubernetes.io/projected/7d6c62d0-83e1-41e7-8780-5dc9be2074a5-kube-api-access-qrqmb\") pod \"insights-runtime-extractor-gdkkk\" (UID: \"7d6c62d0-83e1-41e7-8780-5dc9be2074a5\") " pod="openshift-insights/insights-runtime-extractor-gdkkk" Sep 12 01:45:38.937733 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:38.937718 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7d6c62d0-83e1-41e7-8780-5dc9be2074a5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gdkkk\" (UID: \"7d6c62d0-83e1-41e7-8780-5dc9be2074a5\") " pod="openshift-insights/insights-runtime-extractor-gdkkk" Sep 12 01:45:38.937768 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:38.937749 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7d6c62d0-83e1-41e7-8780-5dc9be2074a5-data-volume\") pod \"insights-runtime-extractor-gdkkk\" (UID: \"7d6c62d0-83e1-41e7-8780-5dc9be2074a5\") " pod="openshift-insights/insights-runtime-extractor-gdkkk" Sep 12 01:45:39.038963 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:39.038933 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7d6c62d0-83e1-41e7-8780-5dc9be2074a5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gdkkk\" (UID: \"7d6c62d0-83e1-41e7-8780-5dc9be2074a5\") " pod="openshift-insights/insights-runtime-extractor-gdkkk" Sep 12 01:45:39.039039 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:39.038985 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7d6c62d0-83e1-41e7-8780-5dc9be2074a5-data-volume\") pod \"insights-runtime-extractor-gdkkk\" (UID: \"7d6c62d0-83e1-41e7-8780-5dc9be2074a5\") " pod="openshift-insights/insights-runtime-extractor-gdkkk" Sep 12 01:45:39.039039 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:39.039016 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7d6c62d0-83e1-41e7-8780-5dc9be2074a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gdkkk\" (UID: \"7d6c62d0-83e1-41e7-8780-5dc9be2074a5\") " pod="openshift-insights/insights-runtime-extractor-gdkkk" Sep 12 01:45:39.039039 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:39.039021 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 12 01:45:39.039160 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:39.039043 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7d6c62d0-83e1-41e7-8780-5dc9be2074a5-crio-socket\") pod \"insights-runtime-extractor-gdkkk\" (UID: \"7d6c62d0-83e1-41e7-8780-5dc9be2074a5\") " pod="openshift-insights/insights-runtime-extractor-gdkkk" Sep 12 01:45:39.039160 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:39.039071 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrqmb\" (UniqueName: \"kubernetes.io/projected/7d6c62d0-83e1-41e7-8780-5dc9be2074a5-kube-api-access-qrqmb\") pod \"insights-runtime-extractor-gdkkk\" (UID: \"7d6c62d0-83e1-41e7-8780-5dc9be2074a5\") " pod="openshift-insights/insights-runtime-extractor-gdkkk" Sep 12 01:45:39.039160 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:39.039095 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7d6c62d0-83e1-41e7-8780-5dc9be2074a5-kube-rbac-proxy-cm podName:7d6c62d0-83e1-41e7-8780-5dc9be2074a5 nodeName:}" failed. No retries permitted until 2026-09-12 01:45:39.539075779 +0000 UTC m=+42.331069565 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7d6c62d0-83e1-41e7-8780-5dc9be2074a5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gdkkk" (UID: "7d6c62d0-83e1-41e7-8780-5dc9be2074a5") : configmap "kube-rbac-proxy" not found Sep 12 01:45:39.039292 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:39.039155 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 12 01:45:39.039292 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:39.039235 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d6c62d0-83e1-41e7-8780-5dc9be2074a5-insights-runtime-extractor-tls podName:7d6c62d0-83e1-41e7-8780-5dc9be2074a5 nodeName:}" failed. No retries permitted until 2026-09-12 01:45:39.539216648 +0000 UTC m=+42.331210422 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7d6c62d0-83e1-41e7-8780-5dc9be2074a5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gdkkk" (UID: "7d6c62d0-83e1-41e7-8780-5dc9be2074a5") : secret "insights-runtime-extractor-tls" not found Sep 12 01:45:39.039380 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:39.039282 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7d6c62d0-83e1-41e7-8780-5dc9be2074a5-crio-socket\") pod \"insights-runtime-extractor-gdkkk\" (UID: \"7d6c62d0-83e1-41e7-8780-5dc9be2074a5\") " pod="openshift-insights/insights-runtime-extractor-gdkkk" Sep 12 01:45:39.039380 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:39.039362 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7d6c62d0-83e1-41e7-8780-5dc9be2074a5-data-volume\") pod \"insights-runtime-extractor-gdkkk\" (UID: \"7d6c62d0-83e1-41e7-8780-5dc9be2074a5\") " pod="openshift-insights/insights-runtime-extractor-gdkkk" Sep 12 01:45:39.048037 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:39.048017 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrqmb\" (UniqueName: \"kubernetes.io/projected/7d6c62d0-83e1-41e7-8780-5dc9be2074a5-kube-api-access-qrqmb\") pod \"insights-runtime-extractor-gdkkk\" (UID: \"7d6c62d0-83e1-41e7-8780-5dc9be2074a5\") " pod="openshift-insights/insights-runtime-extractor-gdkkk" Sep 12 01:45:39.079628 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:39.079569 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:39.182231 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:39.182185 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-bshhm"] Sep 12 01:45:39.182418 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:39.182397 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:39.204749 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:39.204725 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-bshhm"] Sep 12 01:45:39.204860 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:39.204807 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-bshhm" Sep 12 01:45:39.206943 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:39.206924 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Sep 12 01:45:39.207037 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:39.206924 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Sep 12 01:45:39.207096 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:39.207049 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-q585m\"" Sep 12 01:45:39.245465 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:39.245435 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:39.305290 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:39.305264 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:39.341476 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:39.341408 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2d765cc-6edd-4ba0-b8b0-81391dd5aa3d-metrics-tls\") pod \"dns-default-p96nw\" (UID: \"f2d765cc-6edd-4ba0-b8b0-81391dd5aa3d\") " pod="openshift-dns/dns-default-p96nw" Sep 12 01:45:39.341549 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:39.341492 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-registry-tls\") pod \"image-registry-cbdcf5d7-xg922\" (UID: \"5e0ab63f-37d5-4aa5-aeca-4575b5063d13\") " pod="openshift-image-registry/image-registry-cbdcf5d7-xg922" Sep 12 01:45:39.341549 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:39.341528 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl7bl\" (UniqueName: \"kubernetes.io/projected/e083dae5-f950-4f0d-bbc0-4511763aeee2-kube-api-access-rl7bl\") pod \"migrator-8b4d697c5-bshhm\" (UID: \"e083dae5-f950-4f0d-bbc0-4511763aeee2\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-bshhm" Sep 12 01:45:39.341615 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:39.341551 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 12 01:45:39.341650 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:39.341629 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2d765cc-6edd-4ba0-b8b0-81391dd5aa3d-metrics-tls podName:f2d765cc-6edd-4ba0-b8b0-81391dd5aa3d nodeName:}" failed. No retries permitted until 2026-09-12 01:45:47.341608641 +0000 UTC m=+50.133602434 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f2d765cc-6edd-4ba0-b8b0-81391dd5aa3d-metrics-tls") pod "dns-default-p96nw" (UID: "f2d765cc-6edd-4ba0-b8b0-81391dd5aa3d") : secret "dns-default-metrics-tls" not found Sep 12 01:45:39.341650 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:39.341630 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 12 01:45:39.341650 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:39.341647 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cbdcf5d7-xg922: secret "image-registry-tls" not found Sep 12 01:45:39.341743 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:39.341678 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-registry-tls podName:5e0ab63f-37d5-4aa5-aeca-4575b5063d13 nodeName:}" failed. No retries permitted until 2026-09-12 01:45:47.341668433 +0000 UTC m=+50.133662211 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-registry-tls") pod "image-registry-cbdcf5d7-xg922" (UID: "5e0ab63f-37d5-4aa5-aeca-4575b5063d13") : secret "image-registry-tls" not found Sep 12 01:45:39.442058 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:39.442027 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl7bl\" (UniqueName: \"kubernetes.io/projected/e083dae5-f950-4f0d-bbc0-4511763aeee2-kube-api-access-rl7bl\") pod \"migrator-8b4d697c5-bshhm\" (UID: \"e083dae5-f950-4f0d-bbc0-4511763aeee2\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-bshhm" Sep 12 01:45:39.449829 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:39.449803 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl7bl\" (UniqueName: \"kubernetes.io/projected/e083dae5-f950-4f0d-bbc0-4511763aeee2-kube-api-access-rl7bl\") pod \"migrator-8b4d697c5-bshhm\" (UID: \"e083dae5-f950-4f0d-bbc0-4511763aeee2\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-bshhm" Sep 12 01:45:39.513835 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:39.513809 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-bshhm" Sep 12 01:45:39.515669 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:39.515643 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:39.542617 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:39.542593 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7d6c62d0-83e1-41e7-8780-5dc9be2074a5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gdkkk\" (UID: \"7d6c62d0-83e1-41e7-8780-5dc9be2074a5\") " pod="openshift-insights/insights-runtime-extractor-gdkkk" Sep 12 01:45:39.542696 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:39.542645 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7d6c62d0-83e1-41e7-8780-5dc9be2074a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gdkkk\" (UID: \"7d6c62d0-83e1-41e7-8780-5dc9be2074a5\") " pod="openshift-insights/insights-runtime-extractor-gdkkk" Sep 12 01:45:39.542746 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:39.542703 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 12 01:45:39.542785 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:39.542761 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7d6c62d0-83e1-41e7-8780-5dc9be2074a5-kube-rbac-proxy-cm podName:7d6c62d0-83e1-41e7-8780-5dc9be2074a5 nodeName:}" failed. No retries permitted until 2026-09-12 01:45:40.542746819 +0000 UTC m=+43.334740607 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7d6c62d0-83e1-41e7-8780-5dc9be2074a5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gdkkk" (UID: "7d6c62d0-83e1-41e7-8780-5dc9be2074a5") : configmap "kube-rbac-proxy" not found Sep 12 01:45:39.542836 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:39.542789 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 12 01:45:39.542871 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:39.542841 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d6c62d0-83e1-41e7-8780-5dc9be2074a5-insights-runtime-extractor-tls podName:7d6c62d0-83e1-41e7-8780-5dc9be2074a5 nodeName:}" failed. No retries permitted until 2026-09-12 01:45:40.542825766 +0000 UTC m=+43.334819539 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7d6c62d0-83e1-41e7-8780-5dc9be2074a5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gdkkk" (UID: "7d6c62d0-83e1-41e7-8780-5dc9be2074a5") : secret "insights-runtime-extractor-tls" not found Sep 12 01:45:39.794205 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:39.794165 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:39.805971 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:39.805943 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-bshhm"] Sep 12 01:45:39.809703 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:45:39.809677 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode083dae5_f950_4f0d_bbc0_4511763aeee2.slice/crio-fafdd029f914cc025d95c072571ee5e94ad4d6dd72ba5b3f159297c1edf0b981 WatchSource:0}: Error finding container fafdd029f914cc025d95c072571ee5e94ad4d6dd72ba5b3f159297c1edf0b981: Status 404 returned error can't find the container with id fafdd029f914cc025d95c072571ee5e94ad4d6dd72ba5b3f159297c1edf0b981 Sep 12 01:45:39.893441 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:39.893409 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:39.923108 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:39.923081 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-bshhm" event={"ID":"e083dae5-f950-4f0d-bbc0-4511763aeee2","Type":"ContainerStarted","Data":"fafdd029f914cc025d95c072571ee5e94ad4d6dd72ba5b3f159297c1edf0b981"} Sep 12 01:45:40.344600 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:40.344576 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qzpw5_a1140d6e-61e4-4bad-a337-cc28db7dd812/dns-node-resolver/0.log" Sep 12 01:45:40.523091 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:40.523054 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:40.549915 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:40.549880 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7d6c62d0-83e1-41e7-8780-5dc9be2074a5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gdkkk\" (UID: \"7d6c62d0-83e1-41e7-8780-5dc9be2074a5\") " pod="openshift-insights/insights-runtime-extractor-gdkkk" Sep 12 01:45:40.549915 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:40.549922 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7d6c62d0-83e1-41e7-8780-5dc9be2074a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gdkkk\" (UID: \"7d6c62d0-83e1-41e7-8780-5dc9be2074a5\") " pod="openshift-insights/insights-runtime-extractor-gdkkk" Sep 12 01:45:40.550124 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:40.550039 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 12 01:45:40.550124 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:40.550059 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 12 01:45:40.550124 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:40.550120 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7d6c62d0-83e1-41e7-8780-5dc9be2074a5-kube-rbac-proxy-cm podName:7d6c62d0-83e1-41e7-8780-5dc9be2074a5 nodeName:}" failed. No retries permitted until 2026-09-12 01:45:42.550097566 +0000 UTC m=+45.342091354 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7d6c62d0-83e1-41e7-8780-5dc9be2074a5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gdkkk" (UID: "7d6c62d0-83e1-41e7-8780-5dc9be2074a5") : configmap "kube-rbac-proxy" not found Sep 12 01:45:40.550269 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:40.550139 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d6c62d0-83e1-41e7-8780-5dc9be2074a5-insights-runtime-extractor-tls podName:7d6c62d0-83e1-41e7-8780-5dc9be2074a5 nodeName:}" failed. No retries permitted until 2026-09-12 01:45:42.550129719 +0000 UTC m=+45.342123496 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7d6c62d0-83e1-41e7-8780-5dc9be2074a5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gdkkk" (UID: "7d6c62d0-83e1-41e7-8780-5dc9be2074a5") : secret "insights-runtime-extractor-tls" not found Sep 12 01:45:40.624141 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:40.624107 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:41.143999 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:41.143972 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2dpmw_6e1eb253-a93e-4370-9cbd-a5ec2ae013a9/node-ca/0.log" Sep 12 01:45:41.425809 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:41.425787 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:41.526141 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:41.526116 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:41.929726 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:41.929693 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-bshhm" event={"ID":"e083dae5-f950-4f0d-bbc0-4511763aeee2","Type":"ContainerStarted","Data":"8dc14fe2800c5d7fd73bd397867c9db16d994c3c1c9355d0033aaaa654b796a3"} Sep 12 01:45:41.929908 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:41.929750 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-bshhm" event={"ID":"e083dae5-f950-4f0d-bbc0-4511763aeee2","Type":"ContainerStarted","Data":"4b7208b639bfcdeae831533befbc1aac356e9077056f8aa9a9ec5c00fcb5a701"} Sep 12 01:45:41.951066 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:41.951010 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-bshhm" podStartSLOduration=1.464511956 podStartE2EDuration="2.950988173s" totalImagesPullingTime="1.486476217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:45:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:45:41.950345873 +0000 UTC m=+44.742339670" watchObservedRunningTime="2026-09-12 01:45:41.950988173 +0000 UTC m=+44.742981968" Sep 12 01:45:41.953020 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:41.952997 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8bm2n_c84a24ed-159c-4497-863f-03a5a9c0d81d/serve-healthcheck-canary/0.log" Sep 12 01:45:42.304495 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:42.304389 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:42.567584 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:42.567497 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7d6c62d0-83e1-41e7-8780-5dc9be2074a5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gdkkk\" (UID: \"7d6c62d0-83e1-41e7-8780-5dc9be2074a5\") " pod="openshift-insights/insights-runtime-extractor-gdkkk" Sep 12 01:45:42.567584 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:42.567539 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7d6c62d0-83e1-41e7-8780-5dc9be2074a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gdkkk\" (UID: \"7d6c62d0-83e1-41e7-8780-5dc9be2074a5\") " pod="openshift-insights/insights-runtime-extractor-gdkkk" Sep 12 01:45:42.567762 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:42.567632 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 12 01:45:42.567762 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:42.567697 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7d6c62d0-83e1-41e7-8780-5dc9be2074a5-kube-rbac-proxy-cm podName:7d6c62d0-83e1-41e7-8780-5dc9be2074a5 nodeName:}" failed. No retries permitted until 2026-09-12 01:45:46.567680936 +0000 UTC m=+49.359674708 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7d6c62d0-83e1-41e7-8780-5dc9be2074a5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gdkkk" (UID: "7d6c62d0-83e1-41e7-8780-5dc9be2074a5") : configmap "kube-rbac-proxy" not found Sep 12 01:45:42.567762 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:42.567633 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 12 01:45:42.567762 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:42.567761 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d6c62d0-83e1-41e7-8780-5dc9be2074a5-insights-runtime-extractor-tls podName:7d6c62d0-83e1-41e7-8780-5dc9be2074a5 nodeName:}" failed. No retries permitted until 2026-09-12 01:45:46.56775082 +0000 UTC m=+49.359744605 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7d6c62d0-83e1-41e7-8780-5dc9be2074a5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gdkkk" (UID: "7d6c62d0-83e1-41e7-8780-5dc9be2074a5") : secret "insights-runtime-extractor-tls" not found Sep 12 01:45:42.768750 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:42.768710 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/af0906c4-27c2-4ff2-8c04-7f121cc59a3a-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-nnp46\" (UID: \"af0906c4-27c2-4ff2-8c04-7f121cc59a3a\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nnp46" Sep 12 01:45:42.768909 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:42.768844 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 12 01:45:42.768955 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:42.768909 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af0906c4-27c2-4ff2-8c04-7f121cc59a3a-networking-console-plugin-cert podName:af0906c4-27c2-4ff2-8c04-7f121cc59a3a nodeName:}" failed. No retries permitted until 2026-09-12 01:45:50.768893212 +0000 UTC m=+53.560887005 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/af0906c4-27c2-4ff2-8c04-7f121cc59a3a-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-nnp46" (UID: "af0906c4-27c2-4ff2-8c04-7f121cc59a3a") : secret "networking-console-plugin-cert" not found Sep 12 01:45:43.868223 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:43.868190 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:44.996442 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:44.996400 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:46.087465 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:46.087422 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:46.188082 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:46.188052 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:46.355019 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:46.354933 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:46.521813 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:46.521776 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:46.597741 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:46.597713 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7d6c62d0-83e1-41e7-8780-5dc9be2074a5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gdkkk\" (UID: \"7d6c62d0-83e1-41e7-8780-5dc9be2074a5\") " pod="openshift-insights/insights-runtime-extractor-gdkkk" Sep 12 01:45:46.597906 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:46.597750 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7d6c62d0-83e1-41e7-8780-5dc9be2074a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gdkkk\" (UID: \"7d6c62d0-83e1-41e7-8780-5dc9be2074a5\") " pod="openshift-insights/insights-runtime-extractor-gdkkk" Sep 12 01:45:46.597906 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:46.597879 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 12 01:45:46.597976 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:46.597929 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d6c62d0-83e1-41e7-8780-5dc9be2074a5-insights-runtime-extractor-tls podName:7d6c62d0-83e1-41e7-8780-5dc9be2074a5 nodeName:}" failed. No retries permitted until 2026-09-12 01:45:54.597915953 +0000 UTC m=+57.389909739 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7d6c62d0-83e1-41e7-8780-5dc9be2074a5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gdkkk" (UID: "7d6c62d0-83e1-41e7-8780-5dc9be2074a5") : secret "insights-runtime-extractor-tls" not found Sep 12 01:45:46.598205 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:46.598187 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7d6c62d0-83e1-41e7-8780-5dc9be2074a5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gdkkk\" (UID: \"7d6c62d0-83e1-41e7-8780-5dc9be2074a5\") " pod="openshift-insights/insights-runtime-extractor-gdkkk" Sep 12 01:45:46.667666 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:46.667625 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:46.734471 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:46.734434 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:46.886689 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:46.886653 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:47.098274 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:47.098188 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:47.405158 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:47.405126 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-registry-tls\") pod \"image-registry-cbdcf5d7-xg922\" (UID: \"5e0ab63f-37d5-4aa5-aeca-4575b5063d13\") " pod="openshift-image-registry/image-registry-cbdcf5d7-xg922" Sep 12 01:45:47.405330 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:47.405186 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2d765cc-6edd-4ba0-b8b0-81391dd5aa3d-metrics-tls\") pod \"dns-default-p96nw\" (UID: \"f2d765cc-6edd-4ba0-b8b0-81391dd5aa3d\") " pod="openshift-dns/dns-default-p96nw" Sep 12 01:45:47.405330 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:47.405267 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 12 01:45:47.405330 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:47.405291 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cbdcf5d7-xg922: secret "image-registry-tls" not found Sep 12 01:45:47.405330 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:47.405269 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 12 01:45:47.405512 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:47.405345 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2d765cc-6edd-4ba0-b8b0-81391dd5aa3d-metrics-tls podName:f2d765cc-6edd-4ba0-b8b0-81391dd5aa3d nodeName:}" failed. No retries permitted until 2026-09-12 01:46:03.40533198 +0000 UTC m=+66.197325766 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f2d765cc-6edd-4ba0-b8b0-81391dd5aa3d-metrics-tls") pod "dns-default-p96nw" (UID: "f2d765cc-6edd-4ba0-b8b0-81391dd5aa3d") : secret "dns-default-metrics-tls" not found Sep 12 01:45:47.405512 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:47.405357 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-registry-tls podName:5e0ab63f-37d5-4aa5-aeca-4575b5063d13 nodeName:}" failed. No retries permitted until 2026-09-12 01:46:03.405351648 +0000 UTC m=+66.197345420 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-registry-tls") pod "image-registry-cbdcf5d7-xg922" (UID: "5e0ab63f-37d5-4aa5-aeca-4575b5063d13") : secret "image-registry-tls" not found Sep 12 01:45:47.859050 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:47.858966 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:48.058438 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:48.058397 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:48.353335 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:48.353298 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:48.650393 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:48.650360 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:49.054964 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:49.054886 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:49.257892 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:49.257858 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:49.855868 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:49.855829 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:50.054747 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:50.054713 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:50.155442 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:50.155417 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:50.256050 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:50.256016 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:50.657900 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:50.657868 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:50.833132 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:50.833102 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/af0906c4-27c2-4ff2-8c04-7f121cc59a3a-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-nnp46\" (UID: \"af0906c4-27c2-4ff2-8c04-7f121cc59a3a\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nnp46" Sep 12 01:45:50.833296 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:50.833259 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 12 01:45:50.833349 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:45:50.833323 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af0906c4-27c2-4ff2-8c04-7f121cc59a3a-networking-console-plugin-cert podName:af0906c4-27c2-4ff2-8c04-7f121cc59a3a nodeName:}" failed. No retries permitted until 2026-09-12 01:46:06.833307086 +0000 UTC m=+69.625300874 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/af0906c4-27c2-4ff2-8c04-7f121cc59a3a-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-nnp46" (UID: "af0906c4-27c2-4ff2-8c04-7f121cc59a3a") : secret "networking-console-plugin-cert" not found Sep 12 01:45:50.856250 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:50.856212 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:50.903821 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:50.903791 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:51.256107 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:51.256059 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:51.857445 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:51.857411 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:52.450659 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:52.450625 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:52.551818 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:52.551777 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:52.652427 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:52.652398 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:52.703514 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:52.703432 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:52.851430 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:52.851387 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:53.251369 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:53.251329 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:53.853007 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:53.852973 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:54.447919 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:54.447884 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:54.657494 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:54.657438 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:54.662434 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:54.662414 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7d6c62d0-83e1-41e7-8780-5dc9be2074a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gdkkk\" (UID: \"7d6c62d0-83e1-41e7-8780-5dc9be2074a5\") " pod="openshift-insights/insights-runtime-extractor-gdkkk" Sep 12 01:45:54.664801 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:54.664783 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7d6c62d0-83e1-41e7-8780-5dc9be2074a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gdkkk\" (UID: \"7d6c62d0-83e1-41e7-8780-5dc9be2074a5\") " pod="openshift-insights/insights-runtime-extractor-gdkkk" Sep 12 01:45:54.720052 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:54.720007 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gdkkk" Sep 12 01:45:55.020638 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:55.020572 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gdkkk"] Sep 12 01:45:55.023724 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:45:55.023699 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d6c62d0_83e1_41e7_8780_5dc9be2074a5.slice/crio-f600a6b7d923e2a7e7e2e22834476b40e294df118759965396f7c82eb3707320 WatchSource:0}: Error finding container f600a6b7d923e2a7e7e2e22834476b40e294df118759965396f7c82eb3707320: Status 404 returned error can't find the container with id f600a6b7d923e2a7e7e2e22834476b40e294df118759965396f7c82eb3707320 Sep 12 01:45:55.050398 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:55.050374 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:55.253896 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:55.253864 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:55.676810 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:55.676776 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:55.965738 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:55.965646 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gdkkk" event={"ID":"7d6c62d0-83e1-41e7-8780-5dc9be2074a5","Type":"ContainerStarted","Data":"354af9f952b4ddc7e5fd791036ceb8e1593f323a9a10f3f3e217c8d194fdc2e4"} Sep 12 01:45:55.965738 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:55.965709 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gdkkk" event={"ID":"7d6c62d0-83e1-41e7-8780-5dc9be2074a5","Type":"ContainerStarted","Data":"98caf0defb395a2110ebd5b3ca7b62704e784c9535664462eb97eb6262070f70"} Sep 12 01:45:55.965738 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:55.965737 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gdkkk" event={"ID":"7d6c62d0-83e1-41e7-8780-5dc9be2074a5","Type":"ContainerStarted","Data":"f600a6b7d923e2a7e7e2e22834476b40e294df118759965396f7c82eb3707320"} Sep 12 01:45:56.253567 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:56.253480 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:56.657195 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:56.657161 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:56.850116 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:56.850074 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:56.883640 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:56.883614 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mmlsx" Sep 12 01:45:57.052999 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:57.052928 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:45:57.972678 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:45:57.972641 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gdkkk" event={"ID":"7d6c62d0-83e1-41e7-8780-5dc9be2074a5","Type":"ContainerStarted","Data":"789c6d63d0e2ec3ef4b4236fd417913f38dd9077539268e03c60e6eda16ad724"} Sep 12 01:46:02.134550 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:02.134447 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-gdkkk" podStartSLOduration=22.260800105 podStartE2EDuration="24.134425612s" totalImagesPullingTime="1.873625507s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 01:45:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:45:57.98974713 +0000 UTC m=+60.781740923" watchObservedRunningTime="2026-09-12 01:46:02.134425612 +0000 UTC m=+64.926419407" Sep 12 01:46:02.135210 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:02.134788 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-cbdcf5d7-xg922"] Sep 12 01:46:02.135210 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:46:02.135001 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-cbdcf5d7-xg922" podUID="5e0ab63f-37d5-4aa5-aeca-4575b5063d13" Sep 12 01:46:02.242406 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:02.242372 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7dddc48645-jt7vp"] Sep 12 01:46:02.248339 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:02.248324 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7dddc48645-jt7vp" Sep 12 01:46:02.259820 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:02.259797 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:02.263023 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:02.263001 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7dddc48645-jt7vp"] Sep 12 01:46:02.324671 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:02.324631 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6b452077-08b8-4f90-a95a-85e0d51d9e4a-image-registry-private-configuration\") pod \"image-registry-7dddc48645-jt7vp\" (UID: \"6b452077-08b8-4f90-a95a-85e0d51d9e4a\") " pod="openshift-image-registry/image-registry-7dddc48645-jt7vp" Sep 12 01:46:02.324671 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:02.324665 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6b452077-08b8-4f90-a95a-85e0d51d9e4a-trusted-ca\") pod \"image-registry-7dddc48645-jt7vp\" (UID: \"6b452077-08b8-4f90-a95a-85e0d51d9e4a\") " pod="openshift-image-registry/image-registry-7dddc48645-jt7vp" Sep 12 01:46:02.324867 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:02.324685 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6b452077-08b8-4f90-a95a-85e0d51d9e4a-installation-pull-secrets\") pod \"image-registry-7dddc48645-jt7vp\" (UID: \"6b452077-08b8-4f90-a95a-85e0d51d9e4a\") " pod="openshift-image-registry/image-registry-7dddc48645-jt7vp" Sep 12 01:46:02.324867 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:02.324721 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6b452077-08b8-4f90-a95a-85e0d51d9e4a-ca-trust-extracted\") pod \"image-registry-7dddc48645-jt7vp\" (UID: \"6b452077-08b8-4f90-a95a-85e0d51d9e4a\") " pod="openshift-image-registry/image-registry-7dddc48645-jt7vp" Sep 12 01:46:02.324867 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:02.324770 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6b452077-08b8-4f90-a95a-85e0d51d9e4a-registry-certificates\") pod \"image-registry-7dddc48645-jt7vp\" (UID: \"6b452077-08b8-4f90-a95a-85e0d51d9e4a\") " pod="openshift-image-registry/image-registry-7dddc48645-jt7vp" Sep 12 01:46:02.324867 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:02.324807 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96wmw\" (UniqueName: \"kubernetes.io/projected/6b452077-08b8-4f90-a95a-85e0d51d9e4a-kube-api-access-96wmw\") pod \"image-registry-7dddc48645-jt7vp\" (UID: \"6b452077-08b8-4f90-a95a-85e0d51d9e4a\") " pod="openshift-image-registry/image-registry-7dddc48645-jt7vp" Sep 12 01:46:02.324867 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:02.324838 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6b452077-08b8-4f90-a95a-85e0d51d9e4a-bound-sa-token\") pod \"image-registry-7dddc48645-jt7vp\" (UID: \"6b452077-08b8-4f90-a95a-85e0d51d9e4a\") " pod="openshift-image-registry/image-registry-7dddc48645-jt7vp" Sep 12 01:46:02.324867 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:02.324867 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6b452077-08b8-4f90-a95a-85e0d51d9e4a-registry-tls\") pod \"image-registry-7dddc48645-jt7vp\" (UID: \"6b452077-08b8-4f90-a95a-85e0d51d9e4a\") " pod="openshift-image-registry/image-registry-7dddc48645-jt7vp" Sep 12 01:46:02.349713 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:02.349681 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:02.425587 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:02.425525 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6b452077-08b8-4f90-a95a-85e0d51d9e4a-image-registry-private-configuration\") pod \"image-registry-7dddc48645-jt7vp\" (UID: \"6b452077-08b8-4f90-a95a-85e0d51d9e4a\") " pod="openshift-image-registry/image-registry-7dddc48645-jt7vp" Sep 12 01:46:02.425587 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:02.425561 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6b452077-08b8-4f90-a95a-85e0d51d9e4a-trusted-ca\") pod \"image-registry-7dddc48645-jt7vp\" (UID: \"6b452077-08b8-4f90-a95a-85e0d51d9e4a\") " pod="openshift-image-registry/image-registry-7dddc48645-jt7vp" Sep 12 01:46:02.425587 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:02.425579 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6b452077-08b8-4f90-a95a-85e0d51d9e4a-installation-pull-secrets\") pod \"image-registry-7dddc48645-jt7vp\" (UID: \"6b452077-08b8-4f90-a95a-85e0d51d9e4a\") " pod="openshift-image-registry/image-registry-7dddc48645-jt7vp" Sep 12 01:46:02.425761 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:02.425643 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6b452077-08b8-4f90-a95a-85e0d51d9e4a-ca-trust-extracted\") pod \"image-registry-7dddc48645-jt7vp\" (UID: \"6b452077-08b8-4f90-a95a-85e0d51d9e4a\") " pod="openshift-image-registry/image-registry-7dddc48645-jt7vp" Sep 12 01:46:02.425761 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:02.425692 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6b452077-08b8-4f90-a95a-85e0d51d9e4a-registry-certificates\") pod \"image-registry-7dddc48645-jt7vp\" (UID: \"6b452077-08b8-4f90-a95a-85e0d51d9e4a\") " pod="openshift-image-registry/image-registry-7dddc48645-jt7vp" Sep 12 01:46:02.425761 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:02.425721 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96wmw\" (UniqueName: \"kubernetes.io/projected/6b452077-08b8-4f90-a95a-85e0d51d9e4a-kube-api-access-96wmw\") pod \"image-registry-7dddc48645-jt7vp\" (UID: \"6b452077-08b8-4f90-a95a-85e0d51d9e4a\") " pod="openshift-image-registry/image-registry-7dddc48645-jt7vp" Sep 12 01:46:02.425761 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:02.425754 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6b452077-08b8-4f90-a95a-85e0d51d9e4a-bound-sa-token\") pod \"image-registry-7dddc48645-jt7vp\" (UID: \"6b452077-08b8-4f90-a95a-85e0d51d9e4a\") " pod="openshift-image-registry/image-registry-7dddc48645-jt7vp" Sep 12 01:46:02.425946 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:02.425915 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6b452077-08b8-4f90-a95a-85e0d51d9e4a-registry-tls\") pod \"image-registry-7dddc48645-jt7vp\" (UID: \"6b452077-08b8-4f90-a95a-85e0d51d9e4a\") " pod="openshift-image-registry/image-registry-7dddc48645-jt7vp" Sep 12 01:46:02.426147 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:02.426112 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6b452077-08b8-4f90-a95a-85e0d51d9e4a-ca-trust-extracted\") pod \"image-registry-7dddc48645-jt7vp\" (UID: \"6b452077-08b8-4f90-a95a-85e0d51d9e4a\") " pod="openshift-image-registry/image-registry-7dddc48645-jt7vp" Sep 12 01:46:02.426584 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:02.426563 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6b452077-08b8-4f90-a95a-85e0d51d9e4a-trusted-ca\") pod \"image-registry-7dddc48645-jt7vp\" (UID: \"6b452077-08b8-4f90-a95a-85e0d51d9e4a\") " pod="openshift-image-registry/image-registry-7dddc48645-jt7vp" Sep 12 01:46:02.426684 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:02.426619 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6b452077-08b8-4f90-a95a-85e0d51d9e4a-registry-certificates\") pod \"image-registry-7dddc48645-jt7vp\" (UID: \"6b452077-08b8-4f90-a95a-85e0d51d9e4a\") " pod="openshift-image-registry/image-registry-7dddc48645-jt7vp" Sep 12 01:46:02.428069 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:02.428052 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6b452077-08b8-4f90-a95a-85e0d51d9e4a-installation-pull-secrets\") pod \"image-registry-7dddc48645-jt7vp\" (UID: \"6b452077-08b8-4f90-a95a-85e0d51d9e4a\") " pod="openshift-image-registry/image-registry-7dddc48645-jt7vp" Sep 12 01:46:02.428389 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:02.428371 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6b452077-08b8-4f90-a95a-85e0d51d9e4a-registry-tls\") pod \"image-registry-7dddc48645-jt7vp\" (UID: \"6b452077-08b8-4f90-a95a-85e0d51d9e4a\") " pod="openshift-image-registry/image-registry-7dddc48645-jt7vp" Sep 12 01:46:02.428481 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:02.428387 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6b452077-08b8-4f90-a95a-85e0d51d9e4a-image-registry-private-configuration\") pod \"image-registry-7dddc48645-jt7vp\" (UID: \"6b452077-08b8-4f90-a95a-85e0d51d9e4a\") " pod="openshift-image-registry/image-registry-7dddc48645-jt7vp" Sep 12 01:46:02.433657 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:02.433637 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6b452077-08b8-4f90-a95a-85e0d51d9e4a-bound-sa-token\") pod \"image-registry-7dddc48645-jt7vp\" (UID: \"6b452077-08b8-4f90-a95a-85e0d51d9e4a\") " pod="openshift-image-registry/image-registry-7dddc48645-jt7vp" Sep 12 01:46:02.434088 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:02.434071 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96wmw\" (UniqueName: \"kubernetes.io/projected/6b452077-08b8-4f90-a95a-85e0d51d9e4a-kube-api-access-96wmw\") pod \"image-registry-7dddc48645-jt7vp\" (UID: \"6b452077-08b8-4f90-a95a-85e0d51d9e4a\") " pod="openshift-image-registry/image-registry-7dddc48645-jt7vp" Sep 12 01:46:02.451589 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:02.451568 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:02.546183 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:02.546135 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:02.558045 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:02.558024 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-2s9js\"" Sep 12 01:46:02.566809 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:02.566790 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7dddc48645-jt7vp" Sep 12 01:46:02.840902 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:02.840826 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7dddc48645-jt7vp"] Sep 12 01:46:02.844020 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:46:02.843996 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b452077_08b8_4f90_a95a_85e0d51d9e4a.slice/crio-2e36c5a154ec78a4b6a8b465e77c293cad82e07698e4ef93ba82f2562fc91307 WatchSource:0}: Error finding container 2e36c5a154ec78a4b6a8b465e77c293cad82e07698e4ef93ba82f2562fc91307: Status 404 returned error can't find the container with id 2e36c5a154ec78a4b6a8b465e77c293cad82e07698e4ef93ba82f2562fc91307 Sep 12 01:46:02.988264 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:02.988238 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cbdcf5d7-xg922" Sep 12 01:46:02.988264 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:02.988234 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7dddc48645-jt7vp" event={"ID":"6b452077-08b8-4f90-a95a-85e0d51d9e4a","Type":"ContainerStarted","Data":"86dcea840c1dfa92a480886759872ad4779e8d2188760d8e0698d22da7490cf8"} Sep 12 01:46:02.988467 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:02.988302 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7dddc48645-jt7vp" event={"ID":"6b452077-08b8-4f90-a95a-85e0d51d9e4a","Type":"ContainerStarted","Data":"2e36c5a154ec78a4b6a8b465e77c293cad82e07698e4ef93ba82f2562fc91307"} Sep 12 01:46:02.988467 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:02.988336 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7dddc48645-jt7vp" Sep 12 01:46:02.995059 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:02.995028 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cbdcf5d7-xg922" Sep 12 01:46:03.024716 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:03.024663 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7dddc48645-jt7vp" podStartSLOduration=1.024646615 podStartE2EDuration="1.024646615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:46:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:46:03.023434551 +0000 UTC m=+65.815428346" watchObservedRunningTime="2026-09-12 01:46:03.024646615 +0000 UTC m=+65.816640410" Sep 12 01:46:03.029194 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:03.029175 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-installation-pull-secrets\") pod \"5e0ab63f-37d5-4aa5-aeca-4575b5063d13\" (UID: \"5e0ab63f-37d5-4aa5-aeca-4575b5063d13\") " Sep 12 01:46:03.029315 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:03.029201 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-registry-certificates\") pod \"5e0ab63f-37d5-4aa5-aeca-4575b5063d13\" (UID: \"5e0ab63f-37d5-4aa5-aeca-4575b5063d13\") " Sep 12 01:46:03.029315 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:03.029232 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-image-registry-private-configuration\") pod \"5e0ab63f-37d5-4aa5-aeca-4575b5063d13\" (UID: \"5e0ab63f-37d5-4aa5-aeca-4575b5063d13\") " Sep 12 01:46:03.029315 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:03.029250 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-trusted-ca\") pod \"5e0ab63f-37d5-4aa5-aeca-4575b5063d13\" (UID: \"5e0ab63f-37d5-4aa5-aeca-4575b5063d13\") " Sep 12 01:46:03.029315 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:03.029270 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-ca-trust-extracted\") pod \"5e0ab63f-37d5-4aa5-aeca-4575b5063d13\" (UID: \"5e0ab63f-37d5-4aa5-aeca-4575b5063d13\") " Sep 12 01:46:03.029315 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:03.029310 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9f844\" (UniqueName: \"kubernetes.io/projected/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-kube-api-access-9f844\") pod \"5e0ab63f-37d5-4aa5-aeca-4575b5063d13\" (UID: \"5e0ab63f-37d5-4aa5-aeca-4575b5063d13\") " Sep 12 01:46:03.029623 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:03.029340 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-bound-sa-token\") pod \"5e0ab63f-37d5-4aa5-aeca-4575b5063d13\" (UID: \"5e0ab63f-37d5-4aa5-aeca-4575b5063d13\") " Sep 12 01:46:03.030906 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:03.029798 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "5e0ab63f-37d5-4aa5-aeca-4575b5063d13" (UID: "5e0ab63f-37d5-4aa5-aeca-4575b5063d13"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 12 01:46:03.030906 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:03.030821 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "5e0ab63f-37d5-4aa5-aeca-4575b5063d13" (UID: "5e0ab63f-37d5-4aa5-aeca-4575b5063d13"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 12 01:46:03.030906 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:03.030877 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "5e0ab63f-37d5-4aa5-aeca-4575b5063d13" (UID: "5e0ab63f-37d5-4aa5-aeca-4575b5063d13"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 12 01:46:03.031909 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:03.031880 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-kube-api-access-9f844" (OuterVolumeSpecName: "kube-api-access-9f844") pod "5e0ab63f-37d5-4aa5-aeca-4575b5063d13" (UID: "5e0ab63f-37d5-4aa5-aeca-4575b5063d13"). InnerVolumeSpecName "kube-api-access-9f844". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 12 01:46:03.032018 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:03.031886 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "5e0ab63f-37d5-4aa5-aeca-4575b5063d13" (UID: "5e0ab63f-37d5-4aa5-aeca-4575b5063d13"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 12 01:46:03.032070 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:03.032015 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "5e0ab63f-37d5-4aa5-aeca-4575b5063d13" (UID: "5e0ab63f-37d5-4aa5-aeca-4575b5063d13"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 12 01:46:03.032115 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:03.032105 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "5e0ab63f-37d5-4aa5-aeca-4575b5063d13" (UID: "5e0ab63f-37d5-4aa5-aeca-4575b5063d13"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 12 01:46:03.130930 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:03.130902 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9f844\" (UniqueName: \"kubernetes.io/projected/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-kube-api-access-9f844\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 01:46:03.130930 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:03.130929 2579 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-bound-sa-token\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 01:46:03.131097 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:03.130938 2579 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-installation-pull-secrets\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 01:46:03.131097 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:03.130946 2579 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-registry-certificates\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 01:46:03.131097 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:03.130956 2579 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-image-registry-private-configuration\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 01:46:03.131097 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:03.130965 2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-trusted-ca\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 01:46:03.131097 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:03.130974 2579 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-ca-trust-extracted\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 01:46:03.192825 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:03.192783 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:03.433673 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:03.433644 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03-metrics-certs\") pod \"network-metrics-daemon-nhq7s\" (UID: \"b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03\") " pod="openshift-multus/network-metrics-daemon-nhq7s" Sep 12 01:46:03.433817 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:03.433683 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f016f46e-b786-405e-8f6e-faca9e639b5f-original-pull-secret\") pod \"global-pull-secret-syncer-wjmgj\" (UID: \"f016f46e-b786-405e-8f6e-faca9e639b5f\") " pod="kube-system/global-pull-secret-syncer-wjmgj" Sep 12 01:46:03.433817 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:03.433703 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-registry-tls\") pod \"image-registry-cbdcf5d7-xg922\" (UID: \"5e0ab63f-37d5-4aa5-aeca-4575b5063d13\") " pod="openshift-image-registry/image-registry-cbdcf5d7-xg922" Sep 12 01:46:03.433929 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:03.433873 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frwnn\" (UniqueName: \"kubernetes.io/projected/f02f888c-7971-4e66-8a7d-3e180b6807d9-kube-api-access-frwnn\") pod \"network-check-target-bxnbn\" (UID: \"f02f888c-7971-4e66-8a7d-3e180b6807d9\") " pod="openshift-network-diagnostics/network-check-target-bxnbn" Sep 12 01:46:03.433978 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:03.433939 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2d765cc-6edd-4ba0-b8b0-81391dd5aa3d-metrics-tls\") pod \"dns-default-p96nw\" (UID: \"f2d765cc-6edd-4ba0-b8b0-81391dd5aa3d\") " pod="openshift-dns/dns-default-p96nw" Sep 12 01:46:03.436041 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:03.436021 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 12 01:46:03.436172 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:03.436021 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 12 01:46:03.436333 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:03.436315 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-registry-tls\") pod \"image-registry-cbdcf5d7-xg922\" (UID: \"5e0ab63f-37d5-4aa5-aeca-4575b5063d13\") " pod="openshift-image-registry/image-registry-cbdcf5d7-xg922" Sep 12 01:46:03.436410 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:03.436394 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2d765cc-6edd-4ba0-b8b0-81391dd5aa3d-metrics-tls\") pod \"dns-default-p96nw\" (UID: \"f2d765cc-6edd-4ba0-b8b0-81391dd5aa3d\") " pod="openshift-dns/dns-default-p96nw" Sep 12 01:46:03.436837 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:03.436822 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 12 01:46:03.446382 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:03.446355 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 12 01:46:03.446657 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:03.446641 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f016f46e-b786-405e-8f6e-faca9e639b5f-original-pull-secret\") pod \"global-pull-secret-syncer-wjmgj\" (UID: \"f016f46e-b786-405e-8f6e-faca9e639b5f\") " pod="kube-system/global-pull-secret-syncer-wjmgj" Sep 12 01:46:03.446707 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:03.446661 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03-metrics-certs\") pod \"network-metrics-daemon-nhq7s\" (UID: \"b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03\") " pod="openshift-multus/network-metrics-daemon-nhq7s" Sep 12 01:46:03.456560 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:03.456538 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frwnn\" (UniqueName: \"kubernetes.io/projected/f02f888c-7971-4e66-8a7d-3e180b6807d9-kube-api-access-frwnn\") pod \"network-check-target-bxnbn\" (UID: \"f02f888c-7971-4e66-8a7d-3e180b6807d9\") " pod="openshift-network-diagnostics/network-check-target-bxnbn" Sep 12 01:46:03.533936 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:03.533907 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hmzff\"" Sep 12 01:46:03.534142 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:03.534130 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-registry-tls\") pod \"5e0ab63f-37d5-4aa5-aeca-4575b5063d13\" (UID: \"5e0ab63f-37d5-4aa5-aeca-4575b5063d13\") " Sep 12 01:46:03.536081 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:03.536056 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "5e0ab63f-37d5-4aa5-aeca-4575b5063d13" (UID: "5e0ab63f-37d5-4aa5-aeca-4575b5063d13"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 12 01:46:03.540030 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:03.540016 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fqtlc\"" Sep 12 01:46:03.542583 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:03.542571 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bxnbn" Sep 12 01:46:03.543786 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:03.543773 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wjmgj" Sep 12 01:46:03.548536 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:03.548519 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhq7s" Sep 12 01:46:03.576830 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:03.576733 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:03.629642 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:03.629613 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:03.635502 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:03.635478 2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5e0ab63f-37d5-4aa5-aeca-4575b5063d13-registry-tls\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 01:46:03.648440 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:03.648421 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gmqgn\"" Sep 12 01:46:03.657193 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:03.657173 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-p96nw" Sep 12 01:46:03.819583 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:03.819553 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bxnbn"] Sep 12 01:46:03.822259 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:46:03.822236 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf02f888c_7971_4e66_8a7d_3e180b6807d9.slice/crio-589d7e3826a0327cdd56fbae672171097704e10ec98fa3a06cf8e4ddee22cdb9 WatchSource:0}: Error finding container 589d7e3826a0327cdd56fbae672171097704e10ec98fa3a06cf8e4ddee22cdb9: Status 404 returned error can't find the container with id 589d7e3826a0327cdd56fbae672171097704e10ec98fa3a06cf8e4ddee22cdb9 Sep 12 01:46:03.825402 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:03.825387 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wjmgj"] Sep 12 01:46:03.828336 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:03.828313 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nhq7s"] Sep 12 01:46:03.828690 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:46:03.828664 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf016f46e_b786_405e_8f6e_faca9e639b5f.slice/crio-6b9bf70ca6d30405001f88208132f4b2a418dcad877a03b54d8523f3eea9ffe7 WatchSource:0}: Error finding container 6b9bf70ca6d30405001f88208132f4b2a418dcad877a03b54d8523f3eea9ffe7: Status 404 returned error can't find the container with id 6b9bf70ca6d30405001f88208132f4b2a418dcad877a03b54d8523f3eea9ffe7 Sep 12 01:46:03.832244 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:46:03.832225 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb79c3e7e_cd1c_4833_a3cf_78e6bd3b8f03.slice/crio-ef172e925eac54730f8eb504a7b2b1e0b6a8bbdf0da392ff338079e32b5f9714 WatchSource:0}: Error finding container ef172e925eac54730f8eb504a7b2b1e0b6a8bbdf0da392ff338079e32b5f9714: Status 404 returned error can't find the container with id ef172e925eac54730f8eb504a7b2b1e0b6a8bbdf0da392ff338079e32b5f9714 Sep 12 01:46:03.925714 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:03.925684 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-p96nw"] Sep 12 01:46:03.928431 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:46:03.928407 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2d765cc_6edd_4ba0_b8b0_81391dd5aa3d.slice/crio-3c8e5752280be9f303a62eeea1796150c3edd38ad048763aa586786ba53f799b WatchSource:0}: Error finding container 3c8e5752280be9f303a62eeea1796150c3edd38ad048763aa586786ba53f799b: Status 404 returned error can't find the container with id 3c8e5752280be9f303a62eeea1796150c3edd38ad048763aa586786ba53f799b Sep 12 01:46:03.992893 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:03.992812 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nhq7s" event={"ID":"b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03","Type":"ContainerStarted","Data":"ef172e925eac54730f8eb504a7b2b1e0b6a8bbdf0da392ff338079e32b5f9714"} Sep 12 01:46:03.994028 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:03.993997 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bxnbn" event={"ID":"f02f888c-7971-4e66-8a7d-3e180b6807d9","Type":"ContainerStarted","Data":"589d7e3826a0327cdd56fbae672171097704e10ec98fa3a06cf8e4ddee22cdb9"} Sep 12 01:46:03.995071 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:03.995029 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p96nw" event={"ID":"f2d765cc-6edd-4ba0-b8b0-81391dd5aa3d","Type":"ContainerStarted","Data":"3c8e5752280be9f303a62eeea1796150c3edd38ad048763aa586786ba53f799b"} Sep 12 01:46:03.996056 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:03.996036 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wjmgj" event={"ID":"f016f46e-b786-405e-8f6e-faca9e639b5f","Type":"ContainerStarted","Data":"6b9bf70ca6d30405001f88208132f4b2a418dcad877a03b54d8523f3eea9ffe7"} Sep 12 01:46:03.996152 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:03.996082 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cbdcf5d7-xg922" Sep 12 01:46:04.023005 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:04.022979 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-cbdcf5d7-xg922"] Sep 12 01:46:04.030387 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:04.030368 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-cbdcf5d7-xg922"] Sep 12 01:46:05.322797 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:05.322657 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:05.422790 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:05.422713 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:05.518785 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:05.518748 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-q9jhh"] Sep 12 01:46:05.519931 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:05.519897 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:05.522176 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:05.522154 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-q9jhh" Sep 12 01:46:05.524568 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:05.524549 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 12 01:46:05.524568 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:05.524564 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-jw278\"" Sep 12 01:46:05.529020 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:05.528994 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-q9jhh"] Sep 12 01:46:05.616757 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:05.615784 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-mgsgx"] Sep 12 01:46:05.619541 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:05.618913 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-mgsgx" Sep 12 01:46:05.621517 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:05.621494 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 12 01:46:05.621651 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:05.621546 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 12 01:46:05.621651 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:05.621547 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-dcn4h\"" Sep 12 01:46:05.630854 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:05.630828 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-mgsgx"] Sep 12 01:46:05.635333 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:05.635268 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:05.653716 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:05.653683 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c3d9610e-3251-4262-918f-12f28f4aef86-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-q9jhh\" (UID: \"c3d9610e-3251-4262-918f-12f28f4aef86\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-q9jhh" Sep 12 01:46:05.698663 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:05.698570 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:05.727936 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:05.727901 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e0ab63f-37d5-4aa5-aeca-4575b5063d13" path="/var/lib/kubelet/pods/5e0ab63f-37d5-4aa5-aeca-4575b5063d13/volumes" Sep 12 01:46:05.750258 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:05.750218 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:05.754557 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:05.754527 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c3d9610e-3251-4262-918f-12f28f4aef86-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-q9jhh\" (UID: \"c3d9610e-3251-4262-918f-12f28f4aef86\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-q9jhh" Sep 12 01:46:05.754712 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:05.754648 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whb49\" (UniqueName: \"kubernetes.io/projected/9fbbd8af-609e-4eac-927a-0d90c136858b-kube-api-access-whb49\") pod \"downloads-76f99584f4-mgsgx\" (UID: \"9fbbd8af-609e-4eac-927a-0d90c136858b\") " pod="openshift-console/downloads-76f99584f4-mgsgx" Sep 12 01:46:05.754832 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:46:05.754767 2579 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Sep 12 01:46:05.754915 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:46:05.754848 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3d9610e-3251-4262-918f-12f28f4aef86-tls-certificates podName:c3d9610e-3251-4262-918f-12f28f4aef86 nodeName:}" failed. No retries permitted until 2026-09-12 01:46:06.254833195 +0000 UTC m=+69.046826967 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/c3d9610e-3251-4262-918f-12f28f4aef86-tls-certificates") pod "prometheus-operator-admission-webhook-5864cd99d7-q9jhh" (UID: "c3d9610e-3251-4262-918f-12f28f4aef86") : secret "prometheus-operator-admission-webhook-tls" not found Sep 12 01:46:05.855754 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:05.855332 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whb49\" (UniqueName: \"kubernetes.io/projected/9fbbd8af-609e-4eac-927a-0d90c136858b-kube-api-access-whb49\") pod \"downloads-76f99584f4-mgsgx\" (UID: \"9fbbd8af-609e-4eac-927a-0d90c136858b\") " pod="openshift-console/downloads-76f99584f4-mgsgx" Sep 12 01:46:05.870725 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:05.870654 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whb49\" (UniqueName: \"kubernetes.io/projected/9fbbd8af-609e-4eac-927a-0d90c136858b-kube-api-access-whb49\") pod \"downloads-76f99584f4-mgsgx\" (UID: \"9fbbd8af-609e-4eac-927a-0d90c136858b\") " pod="openshift-console/downloads-76f99584f4-mgsgx" Sep 12 01:46:05.900194 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:05.900159 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:05.932108 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:05.932074 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-mgsgx" Sep 12 01:46:06.007824 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:06.007755 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nhq7s" event={"ID":"b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03","Type":"ContainerStarted","Data":"7141ec7b0cd0f38c07191e57d43e4c5e665120ff8ed24b8905c6e0f87cc1b049"} Sep 12 01:46:06.007824 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:06.007806 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nhq7s" event={"ID":"b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03","Type":"ContainerStarted","Data":"3f88e72fb241c05b98871c3aa45dcc50269b90d64b5c2205fb30453e8caeacb7"} Sep 12 01:46:06.024012 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:06.023954 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-nhq7s" podStartSLOduration=67.977088054 podStartE2EDuration="1m9.023932224s" totalImagesPullingTime="1.04684417s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:44:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:46:06.023592354 +0000 UTC m=+68.815586147" watchObservedRunningTime="2026-09-12 01:46:06.023932224 +0000 UTC m=+68.815926020" Sep 12 01:46:06.147998 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:06.147967 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:06.248256 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:06.248192 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:06.258681 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:06.258610 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c3d9610e-3251-4262-918f-12f28f4aef86-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-q9jhh\" (UID: \"c3d9610e-3251-4262-918f-12f28f4aef86\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-q9jhh" Sep 12 01:46:06.261941 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:06.261908 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c3d9610e-3251-4262-918f-12f28f4aef86-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-q9jhh\" (UID: \"c3d9610e-3251-4262-918f-12f28f4aef86\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-q9jhh" Sep 12 01:46:06.302146 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:06.302117 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-mgsgx"] Sep 12 01:46:06.307252 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:46:06.307224 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fbbd8af_609e_4eac_927a_0d90c136858b.slice/crio-94f47dfb7e34d2a0c46e7961db870e70cd2460574827987156876f2119a05f15 WatchSource:0}: Error finding container 94f47dfb7e34d2a0c46e7961db870e70cd2460574827987156876f2119a05f15: Status 404 returned error can't find the container with id 94f47dfb7e34d2a0c46e7961db870e70cd2460574827987156876f2119a05f15 Sep 12 01:46:06.435156 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:06.435120 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-q9jhh" Sep 12 01:46:06.704556 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:06.704524 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:06.863545 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:06.863504 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/af0906c4-27c2-4ff2-8c04-7f121cc59a3a-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-nnp46\" (UID: \"af0906c4-27c2-4ff2-8c04-7f121cc59a3a\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nnp46" Sep 12 01:46:06.866204 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:06.866183 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/af0906c4-27c2-4ff2-8c04-7f121cc59a3a-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-nnp46\" (UID: \"af0906c4-27c2-4ff2-8c04-7f121cc59a3a\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nnp46" Sep 12 01:46:07.014035 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:07.013951 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p96nw" event={"ID":"f2d765cc-6edd-4ba0-b8b0-81391dd5aa3d","Type":"ContainerStarted","Data":"5f1db1c5ab1393030092a2fb07f6a4508f942457802faa287b7a6cf0e3c0c2e4"} Sep 12 01:46:07.015367 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:07.015338 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-mgsgx" event={"ID":"9fbbd8af-609e-4eac-927a-0d90c136858b","Type":"ContainerStarted","Data":"94f47dfb7e34d2a0c46e7961db870e70cd2460574827987156876f2119a05f15"} Sep 12 01:46:07.119035 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:07.119000 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:07.134737 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:07.134714 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-4l6hw\"" Sep 12 01:46:07.142497 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:07.142478 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nnp46" Sep 12 01:46:07.176885 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:07.176856 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:07.303695 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:07.303582 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:07.356170 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:07.356123 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:07.508576 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:07.508543 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:07.609234 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:07.609152 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:07.660015 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:07.659977 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:07.888170 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:07.888130 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:07.993010 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:07.992975 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:08.092604 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:08.092566 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:08.193375 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:08.193297 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:09.025187 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:09.025129 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bxnbn" event={"ID":"f02f888c-7971-4e66-8a7d-3e180b6807d9","Type":"ContainerStarted","Data":"6f4422a8d423e3e20ff6cd8ff548aef524f2fdfacc897db14cd421ca7a34d832"} Sep 12 01:46:09.025554 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:09.025251 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bxnbn" Sep 12 01:46:09.027516 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:09.027491 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p96nw" event={"ID":"f2d765cc-6edd-4ba0-b8b0-81391dd5aa3d","Type":"ContainerStarted","Data":"1b407410c19693df82bc780e94bb5a5e4e0a218687187abf5cd6d3a1a88b426e"} Sep 12 01:46:09.027715 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:09.027699 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-p96nw" Sep 12 01:46:09.029742 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:09.029722 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wjmgj" event={"ID":"f016f46e-b786-405e-8f6e-faca9e639b5f","Type":"ContainerStarted","Data":"dfe3524d217b02acd51e144302f7168711709c93cd8c6bbfd060a1cc60c1af0f"} Sep 12 01:46:09.041038 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:09.040986 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-bxnbn" podStartSLOduration=67.034121385 podStartE2EDuration="1m12.040968416s" totalImagesPullingTime="5.006847031s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:44:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:46:09.040179718 +0000 UTC m=+71.832173491" watchObservedRunningTime="2026-09-12 01:46:09.040968416 +0000 UTC m=+71.832962212" Sep 12 01:46:09.054641 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:09.054541 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-wjmgj" podStartSLOduration=67.040606103 podStartE2EDuration="1m12.054523975s" totalImagesPullingTime="5.013917872s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 01:44:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:46:09.054024242 +0000 UTC m=+71.846018041" watchObservedRunningTime="2026-09-12 01:46:09.054523975 +0000 UTC m=+71.846517769" Sep 12 01:46:09.077135 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:09.077088 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-p96nw" podStartSLOduration=35.810590856 podStartE2EDuration="38.077065445s" totalImagesPullingTime="2.266474589s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:45:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:46:09.071156617 +0000 UTC m=+71.863150426" watchObservedRunningTime="2026-09-12 01:46:09.077065445 +0000 UTC m=+71.869059240" Sep 12 01:46:09.104667 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:09.104633 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-nnp46"] Sep 12 01:46:09.110116 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:09.110089 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-q9jhh"] Sep 12 01:46:09.113217 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:46:09.113191 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3d9610e_3251_4262_918f_12f28f4aef86.slice/crio-8cbfb1194815950f1d0d6b825d8306d5f9c0bd09cf2ed3a1bace67a75f7995c7 WatchSource:0}: Error finding container 8cbfb1194815950f1d0d6b825d8306d5f9c0bd09cf2ed3a1bace67a75f7995c7: Status 404 returned error can't find the container with id 8cbfb1194815950f1d0d6b825d8306d5f9c0bd09cf2ed3a1bace67a75f7995c7 Sep 12 01:46:10.036080 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:10.036039 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-q9jhh" event={"ID":"c3d9610e-3251-4262-918f-12f28f4aef86","Type":"ContainerStarted","Data":"8cbfb1194815950f1d0d6b825d8306d5f9c0bd09cf2ed3a1bace67a75f7995c7"} Sep 12 01:46:10.038376 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:10.038346 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nnp46" event={"ID":"af0906c4-27c2-4ff2-8c04-7f121cc59a3a","Type":"ContainerStarted","Data":"810f7ad871dbf78467617b966d2b05fda040a5c967df361d4ddaf2dc757ef861"} Sep 12 01:46:11.043479 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:11.043429 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-q9jhh" event={"ID":"c3d9610e-3251-4262-918f-12f28f4aef86","Type":"ContainerStarted","Data":"3cdc13eaca9facef1b1dc4657ac924567c0f359ad7fd91c98edcd896d92bf374"} Sep 12 01:46:11.043927 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:11.043507 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-q9jhh" Sep 12 01:46:11.046074 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:11.046046 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nnp46" event={"ID":"af0906c4-27c2-4ff2-8c04-7f121cc59a3a","Type":"ContainerStarted","Data":"7065aa59c6137ca6731799cafe10d37d8b9189088400e9c325a911cd787b7ba9"} Sep 12 01:46:11.049005 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:11.048984 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-q9jhh" Sep 12 01:46:11.058900 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:11.058855 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-q9jhh" podStartSLOduration=4.617232868 podStartE2EDuration="6.058840649s" totalImagesPullingTime="1.441607781s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:46:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:46:11.058025692 +0000 UTC m=+73.850019488" watchObservedRunningTime="2026-09-12 01:46:11.058840649 +0000 UTC m=+73.850834443" Sep 12 01:46:11.087097 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:11.087036 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nnp46" podStartSLOduration=35.641077603 podStartE2EDuration="37.087016938s" totalImagesPullingTime="1.445939335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:45:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:46:11.085881902 +0000 UTC m=+73.877875701" watchObservedRunningTime="2026-09-12 01:46:11.087016938 +0000 UTC m=+73.879010736" Sep 12 01:46:11.185510 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:11.185473 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:11.239296 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:11.239263 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:11.292575 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:11.292537 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:11.567089 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:11.567051 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-m6t8r"] Sep 12 01:46:11.569765 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:11.569737 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-m6t8r" Sep 12 01:46:11.571906 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:11.571875 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 12 01:46:11.572109 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:11.572084 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 12 01:46:11.572227 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:11.572133 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 12 01:46:11.572942 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:11.572921 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 12 01:46:11.573045 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:11.572964 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-6tcmd\"" Sep 12 01:46:11.573045 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:11.572994 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 12 01:46:11.580153 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:11.580132 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-m6t8r"] Sep 12 01:46:11.598386 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:11.598359 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:11.656107 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:11.656064 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:11.699218 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:11.699178 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4d37e287-c2ca-4776-bac4-b52132a2ad57-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-m6t8r\" (UID: \"4d37e287-c2ca-4776-bac4-b52132a2ad57\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-m6t8r" Sep 12 01:46:11.699411 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:11.699291 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4d37e287-c2ca-4776-bac4-b52132a2ad57-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-m6t8r\" (UID: \"4d37e287-c2ca-4776-bac4-b52132a2ad57\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-m6t8r" Sep 12 01:46:11.699411 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:11.699373 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpgx5\" (UniqueName: \"kubernetes.io/projected/4d37e287-c2ca-4776-bac4-b52132a2ad57-kube-api-access-bpgx5\") pod \"prometheus-operator-68bcd7f987-m6t8r\" (UID: \"4d37e287-c2ca-4776-bac4-b52132a2ad57\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-m6t8r" Sep 12 01:46:11.699411 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:11.699401 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d37e287-c2ca-4776-bac4-b52132a2ad57-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-m6t8r\" (UID: \"4d37e287-c2ca-4776-bac4-b52132a2ad57\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-m6t8r" Sep 12 01:46:11.799927 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:11.799893 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4d37e287-c2ca-4776-bac4-b52132a2ad57-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-m6t8r\" (UID: \"4d37e287-c2ca-4776-bac4-b52132a2ad57\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-m6t8r" Sep 12 01:46:11.800095 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:11.799955 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4d37e287-c2ca-4776-bac4-b52132a2ad57-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-m6t8r\" (UID: \"4d37e287-c2ca-4776-bac4-b52132a2ad57\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-m6t8r" Sep 12 01:46:11.800095 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:46:11.800061 2579 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Sep 12 01:46:11.800234 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:46:11.800119 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d37e287-c2ca-4776-bac4-b52132a2ad57-prometheus-operator-tls podName:4d37e287-c2ca-4776-bac4-b52132a2ad57 nodeName:}" failed. No retries permitted until 2026-09-12 01:46:12.300103139 +0000 UTC m=+75.092096915 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/4d37e287-c2ca-4776-bac4-b52132a2ad57-prometheus-operator-tls") pod "prometheus-operator-68bcd7f987-m6t8r" (UID: "4d37e287-c2ca-4776-bac4-b52132a2ad57") : secret "prometheus-operator-tls" not found Sep 12 01:46:11.800234 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:11.800111 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpgx5\" (UniqueName: \"kubernetes.io/projected/4d37e287-c2ca-4776-bac4-b52132a2ad57-kube-api-access-bpgx5\") pod \"prometheus-operator-68bcd7f987-m6t8r\" (UID: \"4d37e287-c2ca-4776-bac4-b52132a2ad57\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-m6t8r" Sep 12 01:46:11.800338 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:11.800252 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d37e287-c2ca-4776-bac4-b52132a2ad57-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-m6t8r\" (UID: \"4d37e287-c2ca-4776-bac4-b52132a2ad57\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-m6t8r" Sep 12 01:46:11.801007 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:11.800981 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d37e287-c2ca-4776-bac4-b52132a2ad57-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-m6t8r\" (UID: \"4d37e287-c2ca-4776-bac4-b52132a2ad57\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-m6t8r" Sep 12 01:46:11.802734 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:11.802706 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4d37e287-c2ca-4776-bac4-b52132a2ad57-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-m6t8r\" (UID: \"4d37e287-c2ca-4776-bac4-b52132a2ad57\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-m6t8r" Sep 12 01:46:11.808998 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:11.808979 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpgx5\" (UniqueName: \"kubernetes.io/projected/4d37e287-c2ca-4776-bac4-b52132a2ad57-kube-api-access-bpgx5\") pod \"prometheus-operator-68bcd7f987-m6t8r\" (UID: \"4d37e287-c2ca-4776-bac4-b52132a2ad57\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-m6t8r" Sep 12 01:46:12.122497 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:12.122442 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:12.224178 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:12.224114 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:12.305373 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:12.305336 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4d37e287-c2ca-4776-bac4-b52132a2ad57-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-m6t8r\" (UID: \"4d37e287-c2ca-4776-bac4-b52132a2ad57\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-m6t8r" Sep 12 01:46:12.305563 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:46:12.305479 2579 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Sep 12 01:46:12.305563 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:46:12.305543 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d37e287-c2ca-4776-bac4-b52132a2ad57-prometheus-operator-tls podName:4d37e287-c2ca-4776-bac4-b52132a2ad57 nodeName:}" failed. No retries permitted until 2026-09-12 01:46:13.305524617 +0000 UTC m=+76.097518395 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/4d37e287-c2ca-4776-bac4-b52132a2ad57-prometheus-operator-tls") pod "prometheus-operator-68bcd7f987-m6t8r" (UID: "4d37e287-c2ca-4776-bac4-b52132a2ad57") : secret "prometheus-operator-tls" not found Sep 12 01:46:12.393767 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:12.393678 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:13.313781 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:13.313743 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4d37e287-c2ca-4776-bac4-b52132a2ad57-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-m6t8r\" (UID: \"4d37e287-c2ca-4776-bac4-b52132a2ad57\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-m6t8r" Sep 12 01:46:13.316656 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:13.316633 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4d37e287-c2ca-4776-bac4-b52132a2ad57-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-m6t8r\" (UID: \"4d37e287-c2ca-4776-bac4-b52132a2ad57\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-m6t8r" Sep 12 01:46:13.380579 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:13.380544 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-m6t8r" Sep 12 01:46:13.657068 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:13.657035 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-m6t8r"] Sep 12 01:46:13.660565 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:46:13.660531 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d37e287_c2ca_4776_bac4_b52132a2ad57.slice/crio-ebfe7c7a0893c892c42ffaca809fb7df1c64c4a98281dba55992e5b93405ddcc WatchSource:0}: Error finding container ebfe7c7a0893c892c42ffaca809fb7df1c64c4a98281dba55992e5b93405ddcc: Status 404 returned error can't find the container with id ebfe7c7a0893c892c42ffaca809fb7df1c64c4a98281dba55992e5b93405ddcc Sep 12 01:46:14.060629 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:14.060542 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-m6t8r" event={"ID":"4d37e287-c2ca-4776-bac4-b52132a2ad57","Type":"ContainerStarted","Data":"ebfe7c7a0893c892c42ffaca809fb7df1c64c4a98281dba55992e5b93405ddcc"} Sep 12 01:46:14.657702 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:14.657647 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:14.719235 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:14.719198 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:14.769116 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:14.769076 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:15.067757 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:15.067679 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-m6t8r" event={"ID":"4d37e287-c2ca-4776-bac4-b52132a2ad57","Type":"ContainerStarted","Data":"ea0d298e81ed9d1ae6bca6df476411d9cfecb4e466a01cfd21ef23fb987672b6"} Sep 12 01:46:15.067941 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:15.067804 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-m6t8r" event={"ID":"4d37e287-c2ca-4776-bac4-b52132a2ad57","Type":"ContainerStarted","Data":"4b2fa1895ab9b9b28f6cda1a055edfe7336ef43451af9836de8cbfa587c16175"} Sep 12 01:46:15.088232 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:15.088160 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-m6t8r" podStartSLOduration=2.957801153 podStartE2EDuration="4.088135256s" totalImagesPullingTime="1.130334103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:46:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:46:15.086975066 +0000 UTC m=+77.878968863" watchObservedRunningTime="2026-09-12 01:46:15.088135256 +0000 UTC m=+77.880129048" Sep 12 01:46:15.101065 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:15.101027 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:15.201516 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:15.201395 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:16.041846 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:16.041812 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-p96nw" Sep 12 01:46:16.678436 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:16.678399 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:16.821914 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:16.821883 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:16.915744 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:16.915710 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:16.931858 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:16.931780 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-cwrwp"] Sep 12 01:46:16.934712 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:16.934689 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cwrwp" Sep 12 01:46:16.937247 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:16.937223 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 12 01:46:16.938002 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:16.937784 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 12 01:46:16.938002 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:16.937857 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 12 01:46:16.938002 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:16.937785 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-wlmkw\"" Sep 12 01:46:16.948889 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:16.948863 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-cwrwp"] Sep 12 01:46:16.957717 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:16.957695 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8nbjq"] Sep 12 01:46:16.959876 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:16.959861 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8nbjq" Sep 12 01:46:16.961949 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:16.961926 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xdtzd\"" Sep 12 01:46:16.962041 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:16.961989 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 12 01:46:16.962041 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:16.962033 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 12 01:46:16.962284 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:16.962267 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 12 01:46:16.995133 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:16.995090 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:17.043942 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:17.043904 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/41d0d41a-0c50-40f7-b01c-99bdfb61cc18-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-cwrwp\" (UID: \"41d0d41a-0c50-40f7-b01c-99bdfb61cc18\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cwrwp" Sep 12 01:46:17.044375 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:17.043952 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f8921497-7f53-4874-ba19-3528fe3f7cd3-node-exporter-wtmp\") pod \"node-exporter-8nbjq\" (UID: \"f8921497-7f53-4874-ba19-3528fe3f7cd3\") " pod="openshift-monitoring/node-exporter-8nbjq" Sep 12 01:46:17.044375 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:17.044015 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f8921497-7f53-4874-ba19-3528fe3f7cd3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8nbjq\" (UID: \"f8921497-7f53-4874-ba19-3528fe3f7cd3\") " pod="openshift-monitoring/node-exporter-8nbjq" Sep 12 01:46:17.044375 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:17.044057 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sk8h\" (UniqueName: \"kubernetes.io/projected/f8921497-7f53-4874-ba19-3528fe3f7cd3-kube-api-access-4sk8h\") pod \"node-exporter-8nbjq\" (UID: \"f8921497-7f53-4874-ba19-3528fe3f7cd3\") " pod="openshift-monitoring/node-exporter-8nbjq" Sep 12 01:46:17.044375 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:17.044082 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f8921497-7f53-4874-ba19-3528fe3f7cd3-sys\") pod \"node-exporter-8nbjq\" (UID: \"f8921497-7f53-4874-ba19-3528fe3f7cd3\") " pod="openshift-monitoring/node-exporter-8nbjq" Sep 12 01:46:17.044375 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:17.044112 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/41d0d41a-0c50-40f7-b01c-99bdfb61cc18-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-cwrwp\" (UID: \"41d0d41a-0c50-40f7-b01c-99bdfb61cc18\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cwrwp" Sep 12 01:46:17.044375 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:17.044195 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/41d0d41a-0c50-40f7-b01c-99bdfb61cc18-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-cwrwp\" (UID: \"41d0d41a-0c50-40f7-b01c-99bdfb61cc18\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cwrwp" Sep 12 01:46:17.044375 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:17.044227 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mpx2\" (UniqueName: \"kubernetes.io/projected/41d0d41a-0c50-40f7-b01c-99bdfb61cc18-kube-api-access-7mpx2\") pod \"kube-state-metrics-59fbdd77d-cwrwp\" (UID: \"41d0d41a-0c50-40f7-b01c-99bdfb61cc18\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cwrwp" Sep 12 01:46:17.044375 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:17.044262 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/41d0d41a-0c50-40f7-b01c-99bdfb61cc18-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-cwrwp\" (UID: \"41d0d41a-0c50-40f7-b01c-99bdfb61cc18\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cwrwp" Sep 12 01:46:17.044375 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:17.044292 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f8921497-7f53-4874-ba19-3528fe3f7cd3-root\") pod \"node-exporter-8nbjq\" (UID: \"f8921497-7f53-4874-ba19-3528fe3f7cd3\") " pod="openshift-monitoring/node-exporter-8nbjq" Sep 12 01:46:17.044375 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:17.044316 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f8921497-7f53-4874-ba19-3528fe3f7cd3-node-exporter-textfile\") pod \"node-exporter-8nbjq\" (UID: \"f8921497-7f53-4874-ba19-3528fe3f7cd3\") " pod="openshift-monitoring/node-exporter-8nbjq" Sep 12 01:46:17.044375 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:17.044335 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41d0d41a-0c50-40f7-b01c-99bdfb61cc18-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-cwrwp\" (UID: \"41d0d41a-0c50-40f7-b01c-99bdfb61cc18\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cwrwp" Sep 12 01:46:17.044375 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:17.044348 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8921497-7f53-4874-ba19-3528fe3f7cd3-metrics-client-ca\") pod \"node-exporter-8nbjq\" (UID: \"f8921497-7f53-4874-ba19-3528fe3f7cd3\") " pod="openshift-monitoring/node-exporter-8nbjq" Sep 12 01:46:17.044375 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:17.044365 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f8921497-7f53-4874-ba19-3528fe3f7cd3-node-exporter-accelerators-collector-config\") pod \"node-exporter-8nbjq\" (UID: \"f8921497-7f53-4874-ba19-3528fe3f7cd3\") " pod="openshift-monitoring/node-exporter-8nbjq" Sep 12 01:46:17.044955 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:17.044471 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f8921497-7f53-4874-ba19-3528fe3f7cd3-node-exporter-tls\") pod \"node-exporter-8nbjq\" (UID: \"f8921497-7f53-4874-ba19-3528fe3f7cd3\") " pod="openshift-monitoring/node-exporter-8nbjq" Sep 12 01:46:17.093300 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:17.093260 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:17.145596 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:17.145562 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/41d0d41a-0c50-40f7-b01c-99bdfb61cc18-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-cwrwp\" (UID: \"41d0d41a-0c50-40f7-b01c-99bdfb61cc18\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cwrwp" Sep 12 01:46:17.145763 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:17.145613 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f8921497-7f53-4874-ba19-3528fe3f7cd3-root\") pod \"node-exporter-8nbjq\" (UID: \"f8921497-7f53-4874-ba19-3528fe3f7cd3\") " pod="openshift-monitoring/node-exporter-8nbjq" Sep 12 01:46:17.145763 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:17.145640 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f8921497-7f53-4874-ba19-3528fe3f7cd3-node-exporter-textfile\") pod \"node-exporter-8nbjq\" (UID: \"f8921497-7f53-4874-ba19-3528fe3f7cd3\") " pod="openshift-monitoring/node-exporter-8nbjq" Sep 12 01:46:17.145763 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:17.145666 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41d0d41a-0c50-40f7-b01c-99bdfb61cc18-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-cwrwp\" (UID: \"41d0d41a-0c50-40f7-b01c-99bdfb61cc18\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cwrwp" Sep 12 01:46:17.145763 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:17.145689 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8921497-7f53-4874-ba19-3528fe3f7cd3-metrics-client-ca\") pod \"node-exporter-8nbjq\" (UID: \"f8921497-7f53-4874-ba19-3528fe3f7cd3\") " pod="openshift-monitoring/node-exporter-8nbjq" Sep 12 01:46:17.145763 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:17.145712 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f8921497-7f53-4874-ba19-3528fe3f7cd3-node-exporter-accelerators-collector-config\") pod \"node-exporter-8nbjq\" (UID: \"f8921497-7f53-4874-ba19-3528fe3f7cd3\") " pod="openshift-monitoring/node-exporter-8nbjq" Sep 12 01:46:17.145763 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:17.145759 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f8921497-7f53-4874-ba19-3528fe3f7cd3-node-exporter-tls\") pod \"node-exporter-8nbjq\" (UID: \"f8921497-7f53-4874-ba19-3528fe3f7cd3\") " pod="openshift-monitoring/node-exporter-8nbjq" Sep 12 01:46:17.146011 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:17.145788 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/41d0d41a-0c50-40f7-b01c-99bdfb61cc18-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-cwrwp\" (UID: \"41d0d41a-0c50-40f7-b01c-99bdfb61cc18\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cwrwp" Sep 12 01:46:17.146011 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:17.145811 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f8921497-7f53-4874-ba19-3528fe3f7cd3-node-exporter-wtmp\") pod \"node-exporter-8nbjq\" (UID: \"f8921497-7f53-4874-ba19-3528fe3f7cd3\") " pod="openshift-monitoring/node-exporter-8nbjq" Sep 12 01:46:17.146011 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:17.145844 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f8921497-7f53-4874-ba19-3528fe3f7cd3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8nbjq\" (UID: \"f8921497-7f53-4874-ba19-3528fe3f7cd3\") " pod="openshift-monitoring/node-exporter-8nbjq" Sep 12 01:46:17.146011 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:17.145876 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sk8h\" (UniqueName: \"kubernetes.io/projected/f8921497-7f53-4874-ba19-3528fe3f7cd3-kube-api-access-4sk8h\") pod \"node-exporter-8nbjq\" (UID: \"f8921497-7f53-4874-ba19-3528fe3f7cd3\") " pod="openshift-monitoring/node-exporter-8nbjq" Sep 12 01:46:17.146011 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:17.145915 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f8921497-7f53-4874-ba19-3528fe3f7cd3-sys\") pod \"node-exporter-8nbjq\" (UID: \"f8921497-7f53-4874-ba19-3528fe3f7cd3\") " pod="openshift-monitoring/node-exporter-8nbjq" Sep 12 01:46:17.146011 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:17.145946 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/41d0d41a-0c50-40f7-b01c-99bdfb61cc18-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-cwrwp\" (UID: \"41d0d41a-0c50-40f7-b01c-99bdfb61cc18\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cwrwp" Sep 12 01:46:17.146011 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:17.145985 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/41d0d41a-0c50-40f7-b01c-99bdfb61cc18-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-cwrwp\" (UID: \"41d0d41a-0c50-40f7-b01c-99bdfb61cc18\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cwrwp" Sep 12 01:46:17.146011 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:17.146009 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mpx2\" (UniqueName: \"kubernetes.io/projected/41d0d41a-0c50-40f7-b01c-99bdfb61cc18-kube-api-access-7mpx2\") pod \"kube-state-metrics-59fbdd77d-cwrwp\" (UID: \"41d0d41a-0c50-40f7-b01c-99bdfb61cc18\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cwrwp" Sep 12 01:46:17.146643 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:17.146615 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/41d0d41a-0c50-40f7-b01c-99bdfb61cc18-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-cwrwp\" (UID: \"41d0d41a-0c50-40f7-b01c-99bdfb61cc18\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cwrwp" Sep 12 01:46:17.146824 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:17.146684 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f8921497-7f53-4874-ba19-3528fe3f7cd3-sys\") pod \"node-exporter-8nbjq\" (UID: \"f8921497-7f53-4874-ba19-3528fe3f7cd3\") " pod="openshift-monitoring/node-exporter-8nbjq" Sep 12 01:46:17.146824 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:17.146745 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f8921497-7f53-4874-ba19-3528fe3f7cd3-node-exporter-wtmp\") pod \"node-exporter-8nbjq\" (UID: \"f8921497-7f53-4874-ba19-3528fe3f7cd3\") " pod="openshift-monitoring/node-exporter-8nbjq" Sep 12 01:46:17.147553 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:17.147226 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f8921497-7f53-4874-ba19-3528fe3f7cd3-root\") pod \"node-exporter-8nbjq\" (UID: \"f8921497-7f53-4874-ba19-3528fe3f7cd3\") " pod="openshift-monitoring/node-exporter-8nbjq" Sep 12 01:46:17.147553 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:17.147305 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/41d0d41a-0c50-40f7-b01c-99bdfb61cc18-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-cwrwp\" (UID: \"41d0d41a-0c50-40f7-b01c-99bdfb61cc18\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cwrwp" Sep 12 01:46:17.147732 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:17.147671 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41d0d41a-0c50-40f7-b01c-99bdfb61cc18-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-cwrwp\" (UID: \"41d0d41a-0c50-40f7-b01c-99bdfb61cc18\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cwrwp" Sep 12 01:46:17.147732 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:17.147674 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8921497-7f53-4874-ba19-3528fe3f7cd3-metrics-client-ca\") pod \"node-exporter-8nbjq\" (UID: \"f8921497-7f53-4874-ba19-3528fe3f7cd3\") " pod="openshift-monitoring/node-exporter-8nbjq" Sep 12 01:46:17.148403 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:17.148229 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f8921497-7f53-4874-ba19-3528fe3f7cd3-node-exporter-accelerators-collector-config\") pod \"node-exporter-8nbjq\" (UID: \"f8921497-7f53-4874-ba19-3528fe3f7cd3\") " pod="openshift-monitoring/node-exporter-8nbjq" Sep 12 01:46:17.148403 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:17.148363 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f8921497-7f53-4874-ba19-3528fe3f7cd3-node-exporter-textfile\") pod \"node-exporter-8nbjq\" (UID: \"f8921497-7f53-4874-ba19-3528fe3f7cd3\") " pod="openshift-monitoring/node-exporter-8nbjq" Sep 12 01:46:17.149615 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:17.149570 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/41d0d41a-0c50-40f7-b01c-99bdfb61cc18-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-cwrwp\" (UID: \"41d0d41a-0c50-40f7-b01c-99bdfb61cc18\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cwrwp" Sep 12 01:46:17.150595 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:17.150252 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f8921497-7f53-4874-ba19-3528fe3f7cd3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8nbjq\" (UID: \"f8921497-7f53-4874-ba19-3528fe3f7cd3\") " pod="openshift-monitoring/node-exporter-8nbjq" Sep 12 01:46:17.150595 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:17.150253 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f8921497-7f53-4874-ba19-3528fe3f7cd3-node-exporter-tls\") pod \"node-exporter-8nbjq\" (UID: \"f8921497-7f53-4874-ba19-3528fe3f7cd3\") " pod="openshift-monitoring/node-exporter-8nbjq" Sep 12 01:46:17.151198 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:17.151175 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/41d0d41a-0c50-40f7-b01c-99bdfb61cc18-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-cwrwp\" (UID: \"41d0d41a-0c50-40f7-b01c-99bdfb61cc18\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cwrwp" Sep 12 01:46:17.154808 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:17.154786 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mpx2\" (UniqueName: \"kubernetes.io/projected/41d0d41a-0c50-40f7-b01c-99bdfb61cc18-kube-api-access-7mpx2\") pod \"kube-state-metrics-59fbdd77d-cwrwp\" (UID: \"41d0d41a-0c50-40f7-b01c-99bdfb61cc18\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cwrwp" Sep 12 01:46:17.156324 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:17.156298 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sk8h\" (UniqueName: \"kubernetes.io/projected/f8921497-7f53-4874-ba19-3528fe3f7cd3-kube-api-access-4sk8h\") pod \"node-exporter-8nbjq\" (UID: \"f8921497-7f53-4874-ba19-3528fe3f7cd3\") " pod="openshift-monitoring/node-exporter-8nbjq" Sep 12 01:46:17.191823 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:17.191734 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:17.246351 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:17.246315 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cwrwp" Sep 12 01:46:17.272163 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:17.272138 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8nbjq" Sep 12 01:46:17.284078 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:46:17.284044 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8921497_7f53_4874_ba19_3528fe3f7cd3.slice/crio-f55578e91cabaf8acbe591ac7cdbf2917328d3b1ccdc1bb563807a4b62d8b5a0 WatchSource:0}: Error finding container f55578e91cabaf8acbe591ac7cdbf2917328d3b1ccdc1bb563807a4b62d8b5a0: Status 404 returned error can't find the container with id f55578e91cabaf8acbe591ac7cdbf2917328d3b1ccdc1bb563807a4b62d8b5a0 Sep 12 01:46:17.295636 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:17.295604 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:17.527076 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:17.526994 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-cwrwp"] Sep 12 01:46:17.531219 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:46:17.531191 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41d0d41a_0c50_40f7_b01c_99bdfb61cc18.slice/crio-3ebc7ea043c8d0731ede61b8e46fba810309a5c7b1b05bf2a6b9f574203d157c WatchSource:0}: Error finding container 3ebc7ea043c8d0731ede61b8e46fba810309a5c7b1b05bf2a6b9f574203d157c: Status 404 returned error can't find the container with id 3ebc7ea043c8d0731ede61b8e46fba810309a5c7b1b05bf2a6b9f574203d157c Sep 12 01:46:17.878651 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:17.878619 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:17.978898 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:17.978858 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:17.986206 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:17.986176 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 12 01:46:17.989367 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:17.989342 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:46:17.991903 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:17.991829 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 12 01:46:17.991903 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:17.991875 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 12 01:46:17.992100 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:17.991829 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 12 01:46:17.992100 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:17.991832 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 12 01:46:17.992198 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:17.992162 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 12 01:46:17.992263 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:17.992239 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-fqfd8\"" Sep 12 01:46:17.992327 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:17.992288 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 12 01:46:17.992383 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:17.992325 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 12 01:46:17.992477 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:17.992440 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 12 01:46:17.992581 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:17.992560 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 12 01:46:18.003325 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:18.003294 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 12 01:46:18.052595 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:18.052556 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8286\" (UniqueName: \"kubernetes.io/projected/c87497e0-9439-45a2-896b-dfa72ea98368-kube-api-access-r8286\") pod \"alertmanager-main-0\" (UID: \"c87497e0-9439-45a2-896b-dfa72ea98368\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:46:18.052595 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:18.052600 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c87497e0-9439-45a2-896b-dfa72ea98368-config-volume\") pod \"alertmanager-main-0\" (UID: \"c87497e0-9439-45a2-896b-dfa72ea98368\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:46:18.053115 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:18.052695 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c87497e0-9439-45a2-896b-dfa72ea98368-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c87497e0-9439-45a2-896b-dfa72ea98368\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:46:18.053115 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:18.052769 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c87497e0-9439-45a2-896b-dfa72ea98368-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c87497e0-9439-45a2-896b-dfa72ea98368\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:46:18.053115 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:18.052799 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c87497e0-9439-45a2-896b-dfa72ea98368-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c87497e0-9439-45a2-896b-dfa72ea98368\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:46:18.053115 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:18.052825 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c87497e0-9439-45a2-896b-dfa72ea98368-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c87497e0-9439-45a2-896b-dfa72ea98368\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:46:18.053115 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:18.052861 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c87497e0-9439-45a2-896b-dfa72ea98368-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c87497e0-9439-45a2-896b-dfa72ea98368\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:46:18.053115 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:18.052887 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c87497e0-9439-45a2-896b-dfa72ea98368-web-config\") pod \"alertmanager-main-0\" (UID: \"c87497e0-9439-45a2-896b-dfa72ea98368\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:46:18.053115 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:18.052918 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c87497e0-9439-45a2-896b-dfa72ea98368-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c87497e0-9439-45a2-896b-dfa72ea98368\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:46:18.053115 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:18.052945 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c87497e0-9439-45a2-896b-dfa72ea98368-config-out\") pod \"alertmanager-main-0\" (UID: \"c87497e0-9439-45a2-896b-dfa72ea98368\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:46:18.053115 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:18.052966 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c87497e0-9439-45a2-896b-dfa72ea98368-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c87497e0-9439-45a2-896b-dfa72ea98368\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:46:18.053115 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:18.053008 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c87497e0-9439-45a2-896b-dfa72ea98368-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c87497e0-9439-45a2-896b-dfa72ea98368\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:46:18.053115 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:18.053034 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c87497e0-9439-45a2-896b-dfa72ea98368-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c87497e0-9439-45a2-896b-dfa72ea98368\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:46:18.055811 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:18.055777 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:18.081095 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:18.081052 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8nbjq" event={"ID":"f8921497-7f53-4874-ba19-3528fe3f7cd3","Type":"ContainerStarted","Data":"f55578e91cabaf8acbe591ac7cdbf2917328d3b1ccdc1bb563807a4b62d8b5a0"} Sep 12 01:46:18.082464 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:18.082420 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cwrwp" event={"ID":"41d0d41a-0c50-40f7-b01c-99bdfb61cc18","Type":"ContainerStarted","Data":"3ebc7ea043c8d0731ede61b8e46fba810309a5c7b1b05bf2a6b9f574203d157c"} Sep 12 01:46:18.154169 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:18.154085 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c87497e0-9439-45a2-896b-dfa72ea98368-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c87497e0-9439-45a2-896b-dfa72ea98368\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:46:18.154169 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:18.154144 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c87497e0-9439-45a2-896b-dfa72ea98368-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c87497e0-9439-45a2-896b-dfa72ea98368\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:46:18.154367 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:18.154177 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8286\" (UniqueName: \"kubernetes.io/projected/c87497e0-9439-45a2-896b-dfa72ea98368-kube-api-access-r8286\") pod \"alertmanager-main-0\" (UID: \"c87497e0-9439-45a2-896b-dfa72ea98368\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:46:18.154367 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:18.154193 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c87497e0-9439-45a2-896b-dfa72ea98368-config-volume\") pod \"alertmanager-main-0\" (UID: \"c87497e0-9439-45a2-896b-dfa72ea98368\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:46:18.154367 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:18.154231 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c87497e0-9439-45a2-896b-dfa72ea98368-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c87497e0-9439-45a2-896b-dfa72ea98368\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:46:18.154367 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:18.154263 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c87497e0-9439-45a2-896b-dfa72ea98368-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c87497e0-9439-45a2-896b-dfa72ea98368\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:46:18.154367 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:18.154283 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c87497e0-9439-45a2-896b-dfa72ea98368-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c87497e0-9439-45a2-896b-dfa72ea98368\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:46:18.154367 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:18.154304 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c87497e0-9439-45a2-896b-dfa72ea98368-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c87497e0-9439-45a2-896b-dfa72ea98368\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:46:18.154367 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:18.154338 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c87497e0-9439-45a2-896b-dfa72ea98368-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c87497e0-9439-45a2-896b-dfa72ea98368\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:46:18.154367 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:18.154364 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c87497e0-9439-45a2-896b-dfa72ea98368-web-config\") pod \"alertmanager-main-0\" (UID: \"c87497e0-9439-45a2-896b-dfa72ea98368\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:46:18.154714 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:18.154393 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c87497e0-9439-45a2-896b-dfa72ea98368-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c87497e0-9439-45a2-896b-dfa72ea98368\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:46:18.154714 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:18.154421 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c87497e0-9439-45a2-896b-dfa72ea98368-config-out\") pod \"alertmanager-main-0\" (UID: \"c87497e0-9439-45a2-896b-dfa72ea98368\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:46:18.154714 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:18.154438 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c87497e0-9439-45a2-896b-dfa72ea98368-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c87497e0-9439-45a2-896b-dfa72ea98368\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:46:18.156381 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:18.155653 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c87497e0-9439-45a2-896b-dfa72ea98368-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c87497e0-9439-45a2-896b-dfa72ea98368\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:46:18.156381 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:18.156317 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c87497e0-9439-45a2-896b-dfa72ea98368-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c87497e0-9439-45a2-896b-dfa72ea98368\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:46:18.157252 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:46:18.156671 2579 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Sep 12 01:46:18.157252 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:46:18.156744 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c87497e0-9439-45a2-896b-dfa72ea98368-secret-alertmanager-main-tls podName:c87497e0-9439-45a2-896b-dfa72ea98368 nodeName:}" failed. No retries permitted until 2026-09-12 01:46:18.656721283 +0000 UTC m=+81.448715058 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/c87497e0-9439-45a2-896b-dfa72ea98368-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "c87497e0-9439-45a2-896b-dfa72ea98368") : secret "alertmanager-main-tls" not found Sep 12 01:46:18.157252 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:18.157202 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c87497e0-9439-45a2-896b-dfa72ea98368-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c87497e0-9439-45a2-896b-dfa72ea98368\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:46:18.159615 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:18.159588 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c87497e0-9439-45a2-896b-dfa72ea98368-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c87497e0-9439-45a2-896b-dfa72ea98368\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:46:18.159756 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:18.159733 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c87497e0-9439-45a2-896b-dfa72ea98368-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c87497e0-9439-45a2-896b-dfa72ea98368\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:46:18.160059 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:18.160029 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c87497e0-9439-45a2-896b-dfa72ea98368-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c87497e0-9439-45a2-896b-dfa72ea98368\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:46:18.160349 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:18.160314 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c87497e0-9439-45a2-896b-dfa72ea98368-config-volume\") pod \"alertmanager-main-0\" (UID: \"c87497e0-9439-45a2-896b-dfa72ea98368\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:46:18.160693 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:18.160667 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c87497e0-9439-45a2-896b-dfa72ea98368-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c87497e0-9439-45a2-896b-dfa72ea98368\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:46:18.160840 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:18.160817 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c87497e0-9439-45a2-896b-dfa72ea98368-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c87497e0-9439-45a2-896b-dfa72ea98368\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:46:18.161051 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:18.161027 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c87497e0-9439-45a2-896b-dfa72ea98368-web-config\") pod \"alertmanager-main-0\" (UID: \"c87497e0-9439-45a2-896b-dfa72ea98368\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:46:18.161311 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:18.161288 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c87497e0-9439-45a2-896b-dfa72ea98368-config-out\") pod \"alertmanager-main-0\" (UID: \"c87497e0-9439-45a2-896b-dfa72ea98368\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:46:18.163651 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:18.163629 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8286\" (UniqueName: \"kubernetes.io/projected/c87497e0-9439-45a2-896b-dfa72ea98368-kube-api-access-r8286\") pod \"alertmanager-main-0\" (UID: \"c87497e0-9439-45a2-896b-dfa72ea98368\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:46:18.328914 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:18.328864 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:18.658611 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:18.658573 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c87497e0-9439-45a2-896b-dfa72ea98368-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c87497e0-9439-45a2-896b-dfa72ea98368\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:46:18.661427 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:18.661402 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c87497e0-9439-45a2-896b-dfa72ea98368-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c87497e0-9439-45a2-896b-dfa72ea98368\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:46:18.901220 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:18.901092 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:46:19.095832 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:19.095749 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:19.197225 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:19.197193 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:19.841131 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:19.841098 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:19.906962 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:19.906924 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:19.911611 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:19.911586 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-f67ddbc77-l6kn6"] Sep 12 01:46:19.915191 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:19.915167 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-f67ddbc77-l6kn6" Sep 12 01:46:19.917863 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:19.917836 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 12 01:46:19.917979 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:19.917898 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-6l5b8\"" Sep 12 01:46:19.918195 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:19.918043 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-c6svglchsbu7i\"" Sep 12 01:46:19.918195 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:19.918103 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 12 01:46:19.918695 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:19.918677 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 12 01:46:19.918894 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:19.918878 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 12 01:46:19.919067 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:19.919046 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 12 01:46:19.927863 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:19.927844 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-f67ddbc77-l6kn6"] Sep 12 01:46:20.072447 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:20.072402 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7905e746-9f1b-41cb-af97-43f87dc75594-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-f67ddbc77-l6kn6\" (UID: \"7905e746-9f1b-41cb-af97-43f87dc75594\") " pod="openshift-monitoring/thanos-querier-f67ddbc77-l6kn6" Sep 12 01:46:20.072668 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:20.072481 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7905e746-9f1b-41cb-af97-43f87dc75594-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-f67ddbc77-l6kn6\" (UID: \"7905e746-9f1b-41cb-af97-43f87dc75594\") " pod="openshift-monitoring/thanos-querier-f67ddbc77-l6kn6" Sep 12 01:46:20.072668 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:20.072529 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7905e746-9f1b-41cb-af97-43f87dc75594-secret-grpc-tls\") pod \"thanos-querier-f67ddbc77-l6kn6\" (UID: \"7905e746-9f1b-41cb-af97-43f87dc75594\") " pod="openshift-monitoring/thanos-querier-f67ddbc77-l6kn6" Sep 12 01:46:20.072668 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:20.072567 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7905e746-9f1b-41cb-af97-43f87dc75594-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-f67ddbc77-l6kn6\" (UID: \"7905e746-9f1b-41cb-af97-43f87dc75594\") " pod="openshift-monitoring/thanos-querier-f67ddbc77-l6kn6" Sep 12 01:46:20.072668 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:20.072591 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7905e746-9f1b-41cb-af97-43f87dc75594-secret-thanos-querier-tls\") pod \"thanos-querier-f67ddbc77-l6kn6\" (UID: \"7905e746-9f1b-41cb-af97-43f87dc75594\") " pod="openshift-monitoring/thanos-querier-f67ddbc77-l6kn6" Sep 12 01:46:20.072668 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:20.072658 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7905e746-9f1b-41cb-af97-43f87dc75594-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-f67ddbc77-l6kn6\" (UID: \"7905e746-9f1b-41cb-af97-43f87dc75594\") " pod="openshift-monitoring/thanos-querier-f67ddbc77-l6kn6" Sep 12 01:46:20.072900 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:20.072692 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4lz4\" (UniqueName: \"kubernetes.io/projected/7905e746-9f1b-41cb-af97-43f87dc75594-kube-api-access-x4lz4\") pod \"thanos-querier-f67ddbc77-l6kn6\" (UID: \"7905e746-9f1b-41cb-af97-43f87dc75594\") " pod="openshift-monitoring/thanos-querier-f67ddbc77-l6kn6" Sep 12 01:46:20.072900 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:20.072732 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7905e746-9f1b-41cb-af97-43f87dc75594-metrics-client-ca\") pod \"thanos-querier-f67ddbc77-l6kn6\" (UID: \"7905e746-9f1b-41cb-af97-43f87dc75594\") " pod="openshift-monitoring/thanos-querier-f67ddbc77-l6kn6" Sep 12 01:46:20.107067 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:20.106976 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:20.174102 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:20.174066 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7905e746-9f1b-41cb-af97-43f87dc75594-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-f67ddbc77-l6kn6\" (UID: \"7905e746-9f1b-41cb-af97-43f87dc75594\") " pod="openshift-monitoring/thanos-querier-f67ddbc77-l6kn6" Sep 12 01:46:20.174277 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:20.174112 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7905e746-9f1b-41cb-af97-43f87dc75594-secret-thanos-querier-tls\") pod \"thanos-querier-f67ddbc77-l6kn6\" (UID: \"7905e746-9f1b-41cb-af97-43f87dc75594\") " pod="openshift-monitoring/thanos-querier-f67ddbc77-l6kn6" Sep 12 01:46:20.174277 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:20.174151 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7905e746-9f1b-41cb-af97-43f87dc75594-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-f67ddbc77-l6kn6\" (UID: \"7905e746-9f1b-41cb-af97-43f87dc75594\") " pod="openshift-monitoring/thanos-querier-f67ddbc77-l6kn6" Sep 12 01:46:20.174277 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:20.174176 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4lz4\" (UniqueName: \"kubernetes.io/projected/7905e746-9f1b-41cb-af97-43f87dc75594-kube-api-access-x4lz4\") pod \"thanos-querier-f67ddbc77-l6kn6\" (UID: \"7905e746-9f1b-41cb-af97-43f87dc75594\") " pod="openshift-monitoring/thanos-querier-f67ddbc77-l6kn6" Sep 12 01:46:20.174277 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:20.174208 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7905e746-9f1b-41cb-af97-43f87dc75594-metrics-client-ca\") pod \"thanos-querier-f67ddbc77-l6kn6\" (UID: \"7905e746-9f1b-41cb-af97-43f87dc75594\") " pod="openshift-monitoring/thanos-querier-f67ddbc77-l6kn6" Sep 12 01:46:20.174277 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:20.174252 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7905e746-9f1b-41cb-af97-43f87dc75594-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-f67ddbc77-l6kn6\" (UID: \"7905e746-9f1b-41cb-af97-43f87dc75594\") " pod="openshift-monitoring/thanos-querier-f67ddbc77-l6kn6" Sep 12 01:46:20.174566 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:20.174288 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7905e746-9f1b-41cb-af97-43f87dc75594-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-f67ddbc77-l6kn6\" (UID: \"7905e746-9f1b-41cb-af97-43f87dc75594\") " pod="openshift-monitoring/thanos-querier-f67ddbc77-l6kn6" Sep 12 01:46:20.174566 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:20.174337 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7905e746-9f1b-41cb-af97-43f87dc75594-secret-grpc-tls\") pod \"thanos-querier-f67ddbc77-l6kn6\" (UID: \"7905e746-9f1b-41cb-af97-43f87dc75594\") " pod="openshift-monitoring/thanos-querier-f67ddbc77-l6kn6" Sep 12 01:46:20.175433 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:20.175405 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7905e746-9f1b-41cb-af97-43f87dc75594-metrics-client-ca\") pod \"thanos-querier-f67ddbc77-l6kn6\" (UID: \"7905e746-9f1b-41cb-af97-43f87dc75594\") " pod="openshift-monitoring/thanos-querier-f67ddbc77-l6kn6" Sep 12 01:46:20.177927 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:20.177897 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7905e746-9f1b-41cb-af97-43f87dc75594-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-f67ddbc77-l6kn6\" (UID: \"7905e746-9f1b-41cb-af97-43f87dc75594\") " pod="openshift-monitoring/thanos-querier-f67ddbc77-l6kn6" Sep 12 01:46:20.178058 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:20.178028 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7905e746-9f1b-41cb-af97-43f87dc75594-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-f67ddbc77-l6kn6\" (UID: \"7905e746-9f1b-41cb-af97-43f87dc75594\") " pod="openshift-monitoring/thanos-querier-f67ddbc77-l6kn6" Sep 12 01:46:20.178541 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:20.178497 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7905e746-9f1b-41cb-af97-43f87dc75594-secret-grpc-tls\") pod \"thanos-querier-f67ddbc77-l6kn6\" (UID: \"7905e746-9f1b-41cb-af97-43f87dc75594\") " pod="openshift-monitoring/thanos-querier-f67ddbc77-l6kn6" Sep 12 01:46:20.178697 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:20.178677 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7905e746-9f1b-41cb-af97-43f87dc75594-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-f67ddbc77-l6kn6\" (UID: \"7905e746-9f1b-41cb-af97-43f87dc75594\") " pod="openshift-monitoring/thanos-querier-f67ddbc77-l6kn6" Sep 12 01:46:20.178977 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:20.178954 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7905e746-9f1b-41cb-af97-43f87dc75594-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-f67ddbc77-l6kn6\" (UID: \"7905e746-9f1b-41cb-af97-43f87dc75594\") " pod="openshift-monitoring/thanos-querier-f67ddbc77-l6kn6" Sep 12 01:46:20.179347 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:20.179321 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7905e746-9f1b-41cb-af97-43f87dc75594-secret-thanos-querier-tls\") pod \"thanos-querier-f67ddbc77-l6kn6\" (UID: \"7905e746-9f1b-41cb-af97-43f87dc75594\") " pod="openshift-monitoring/thanos-querier-f67ddbc77-l6kn6" Sep 12 01:46:20.182146 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:20.182122 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4lz4\" (UniqueName: \"kubernetes.io/projected/7905e746-9f1b-41cb-af97-43f87dc75594-kube-api-access-x4lz4\") pod \"thanos-querier-f67ddbc77-l6kn6\" (UID: \"7905e746-9f1b-41cb-af97-43f87dc75594\") " pod="openshift-monitoring/thanos-querier-f67ddbc77-l6kn6" Sep 12 01:46:20.208052 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:20.207957 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:20.228334 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:20.228082 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-f67ddbc77-l6kn6" Sep 12 01:46:20.737427 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:20.737391 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8475968459-hkw8j"] Sep 12 01:46:20.739868 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:20.739846 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8475968459-hkw8j" Sep 12 01:46:20.742502 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:20.742252 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 12 01:46:20.742502 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:20.742273 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-5x28t\"" Sep 12 01:46:20.742502 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:20.742327 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 12 01:46:20.742926 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:20.742901 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 12 01:46:20.743153 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:20.743125 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 12 01:46:20.743604 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:20.743589 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 12 01:46:20.750772 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:20.750738 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:20.750966 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:20.750944 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8475968459-hkw8j"] Sep 12 01:46:20.751736 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:20.751714 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 12 01:46:20.880580 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:20.880544 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/511982d8-0ade-4304-8d0d-ccb1f951fcb2-console-config\") pod \"console-8475968459-hkw8j\" (UID: \"511982d8-0ade-4304-8d0d-ccb1f951fcb2\") " pod="openshift-console/console-8475968459-hkw8j" Sep 12 01:46:20.880580 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:20.880584 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/511982d8-0ade-4304-8d0d-ccb1f951fcb2-service-ca\") pod \"console-8475968459-hkw8j\" (UID: \"511982d8-0ade-4304-8d0d-ccb1f951fcb2\") " pod="openshift-console/console-8475968459-hkw8j" Sep 12 01:46:20.880816 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:20.880613 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/511982d8-0ade-4304-8d0d-ccb1f951fcb2-console-serving-cert\") pod \"console-8475968459-hkw8j\" (UID: \"511982d8-0ade-4304-8d0d-ccb1f951fcb2\") " pod="openshift-console/console-8475968459-hkw8j" Sep 12 01:46:20.880816 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:20.880637 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/511982d8-0ade-4304-8d0d-ccb1f951fcb2-oauth-serving-cert\") pod \"console-8475968459-hkw8j\" (UID: \"511982d8-0ade-4304-8d0d-ccb1f951fcb2\") " pod="openshift-console/console-8475968459-hkw8j" Sep 12 01:46:20.880816 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:20.880706 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqxjz\" (UniqueName: \"kubernetes.io/projected/511982d8-0ade-4304-8d0d-ccb1f951fcb2-kube-api-access-jqxjz\") pod \"console-8475968459-hkw8j\" (UID: \"511982d8-0ade-4304-8d0d-ccb1f951fcb2\") " pod="openshift-console/console-8475968459-hkw8j" Sep 12 01:46:20.880816 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:20.880768 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/511982d8-0ade-4304-8d0d-ccb1f951fcb2-console-oauth-config\") pod \"console-8475968459-hkw8j\" (UID: \"511982d8-0ade-4304-8d0d-ccb1f951fcb2\") " pod="openshift-console/console-8475968459-hkw8j" Sep 12 01:46:20.880980 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:20.880825 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/511982d8-0ade-4304-8d0d-ccb1f951fcb2-trusted-ca-bundle\") pod \"console-8475968459-hkw8j\" (UID: \"511982d8-0ade-4304-8d0d-ccb1f951fcb2\") " pod="openshift-console/console-8475968459-hkw8j" Sep 12 01:46:20.981954 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:20.981914 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/511982d8-0ade-4304-8d0d-ccb1f951fcb2-console-oauth-config\") pod \"console-8475968459-hkw8j\" (UID: \"511982d8-0ade-4304-8d0d-ccb1f951fcb2\") " pod="openshift-console/console-8475968459-hkw8j" Sep 12 01:46:20.982122 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:20.981997 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/511982d8-0ade-4304-8d0d-ccb1f951fcb2-trusted-ca-bundle\") pod \"console-8475968459-hkw8j\" (UID: \"511982d8-0ade-4304-8d0d-ccb1f951fcb2\") " pod="openshift-console/console-8475968459-hkw8j" Sep 12 01:46:20.982122 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:20.982049 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/511982d8-0ade-4304-8d0d-ccb1f951fcb2-console-config\") pod \"console-8475968459-hkw8j\" (UID: \"511982d8-0ade-4304-8d0d-ccb1f951fcb2\") " pod="openshift-console/console-8475968459-hkw8j" Sep 12 01:46:20.982122 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:20.982073 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/511982d8-0ade-4304-8d0d-ccb1f951fcb2-service-ca\") pod \"console-8475968459-hkw8j\" (UID: \"511982d8-0ade-4304-8d0d-ccb1f951fcb2\") " pod="openshift-console/console-8475968459-hkw8j" Sep 12 01:46:20.982122 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:20.982099 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/511982d8-0ade-4304-8d0d-ccb1f951fcb2-console-serving-cert\") pod \"console-8475968459-hkw8j\" (UID: \"511982d8-0ade-4304-8d0d-ccb1f951fcb2\") " pod="openshift-console/console-8475968459-hkw8j" Sep 12 01:46:20.982308 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:20.982123 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/511982d8-0ade-4304-8d0d-ccb1f951fcb2-oauth-serving-cert\") pod \"console-8475968459-hkw8j\" (UID: \"511982d8-0ade-4304-8d0d-ccb1f951fcb2\") " pod="openshift-console/console-8475968459-hkw8j" Sep 12 01:46:20.982308 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:20.982162 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqxjz\" (UniqueName: \"kubernetes.io/projected/511982d8-0ade-4304-8d0d-ccb1f951fcb2-kube-api-access-jqxjz\") pod \"console-8475968459-hkw8j\" (UID: \"511982d8-0ade-4304-8d0d-ccb1f951fcb2\") " pod="openshift-console/console-8475968459-hkw8j" Sep 12 01:46:20.983064 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:20.982892 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/511982d8-0ade-4304-8d0d-ccb1f951fcb2-service-ca\") pod \"console-8475968459-hkw8j\" (UID: \"511982d8-0ade-4304-8d0d-ccb1f951fcb2\") " pod="openshift-console/console-8475968459-hkw8j" Sep 12 01:46:20.983064 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:20.982966 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/511982d8-0ade-4304-8d0d-ccb1f951fcb2-trusted-ca-bundle\") pod \"console-8475968459-hkw8j\" (UID: \"511982d8-0ade-4304-8d0d-ccb1f951fcb2\") " pod="openshift-console/console-8475968459-hkw8j" Sep 12 01:46:20.983064 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:20.982990 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/511982d8-0ade-4304-8d0d-ccb1f951fcb2-oauth-serving-cert\") pod \"console-8475968459-hkw8j\" (UID: \"511982d8-0ade-4304-8d0d-ccb1f951fcb2\") " pod="openshift-console/console-8475968459-hkw8j" Sep 12 01:46:20.983284 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:20.983134 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/511982d8-0ade-4304-8d0d-ccb1f951fcb2-console-config\") pod \"console-8475968459-hkw8j\" (UID: \"511982d8-0ade-4304-8d0d-ccb1f951fcb2\") " pod="openshift-console/console-8475968459-hkw8j" Sep 12 01:46:20.984979 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:20.984954 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/511982d8-0ade-4304-8d0d-ccb1f951fcb2-console-oauth-config\") pod \"console-8475968459-hkw8j\" (UID: \"511982d8-0ade-4304-8d0d-ccb1f951fcb2\") " pod="openshift-console/console-8475968459-hkw8j" Sep 12 01:46:20.985129 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:20.985103 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/511982d8-0ade-4304-8d0d-ccb1f951fcb2-console-serving-cert\") pod \"console-8475968459-hkw8j\" (UID: \"511982d8-0ade-4304-8d0d-ccb1f951fcb2\") " pod="openshift-console/console-8475968459-hkw8j" Sep 12 01:46:20.994387 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:20.994326 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqxjz\" (UniqueName: \"kubernetes.io/projected/511982d8-0ade-4304-8d0d-ccb1f951fcb2-kube-api-access-jqxjz\") pod \"console-8475968459-hkw8j\" (UID: \"511982d8-0ade-4304-8d0d-ccb1f951fcb2\") " pod="openshift-console/console-8475968459-hkw8j" Sep 12 01:46:21.060603 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:21.060566 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8475968459-hkw8j" Sep 12 01:46:21.227384 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:21.227349 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6ff6c7b584-hbbsv"] Sep 12 01:46:21.229688 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:21.229669 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6ff6c7b584-hbbsv" Sep 12 01:46:21.232061 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:21.232034 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 12 01:46:21.232201 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:21.232164 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 12 01:46:21.233311 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:21.233275 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 12 01:46:21.233426 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:21.233319 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 12 01:46:21.233426 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:21.233338 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-nq2zp\"" Sep 12 01:46:21.234606 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:21.234584 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-d6h5qsgig3bvc\"" Sep 12 01:46:21.239533 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:21.239506 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6ff6c7b584-hbbsv"] Sep 12 01:46:21.386037 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:21.386001 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/01af4e11-0c36-4ddb-afa6-79acbc8ec2c1-secret-metrics-server-tls\") pod \"metrics-server-6ff6c7b584-hbbsv\" (UID: \"01af4e11-0c36-4ddb-afa6-79acbc8ec2c1\") " pod="openshift-monitoring/metrics-server-6ff6c7b584-hbbsv" Sep 12 01:46:21.386230 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:21.386055 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzt6s\" (UniqueName: \"kubernetes.io/projected/01af4e11-0c36-4ddb-afa6-79acbc8ec2c1-kube-api-access-kzt6s\") pod \"metrics-server-6ff6c7b584-hbbsv\" (UID: \"01af4e11-0c36-4ddb-afa6-79acbc8ec2c1\") " pod="openshift-monitoring/metrics-server-6ff6c7b584-hbbsv" Sep 12 01:46:21.386230 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:21.386117 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/01af4e11-0c36-4ddb-afa6-79acbc8ec2c1-audit-log\") pod \"metrics-server-6ff6c7b584-hbbsv\" (UID: \"01af4e11-0c36-4ddb-afa6-79acbc8ec2c1\") " pod="openshift-monitoring/metrics-server-6ff6c7b584-hbbsv" Sep 12 01:46:21.386230 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:21.386133 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/01af4e11-0c36-4ddb-afa6-79acbc8ec2c1-client-ca-bundle\") pod \"metrics-server-6ff6c7b584-hbbsv\" (UID: \"01af4e11-0c36-4ddb-afa6-79acbc8ec2c1\") " pod="openshift-monitoring/metrics-server-6ff6c7b584-hbbsv" Sep 12 01:46:21.386230 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:21.386165 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01af4e11-0c36-4ddb-afa6-79acbc8ec2c1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6ff6c7b584-hbbsv\" (UID: \"01af4e11-0c36-4ddb-afa6-79acbc8ec2c1\") " pod="openshift-monitoring/metrics-server-6ff6c7b584-hbbsv" Sep 12 01:46:21.386230 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:21.386193 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/01af4e11-0c36-4ddb-afa6-79acbc8ec2c1-secret-metrics-server-client-certs\") pod \"metrics-server-6ff6c7b584-hbbsv\" (UID: \"01af4e11-0c36-4ddb-afa6-79acbc8ec2c1\") " pod="openshift-monitoring/metrics-server-6ff6c7b584-hbbsv" Sep 12 01:46:21.386496 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:21.386301 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/01af4e11-0c36-4ddb-afa6-79acbc8ec2c1-metrics-server-audit-profiles\") pod \"metrics-server-6ff6c7b584-hbbsv\" (UID: \"01af4e11-0c36-4ddb-afa6-79acbc8ec2c1\") " pod="openshift-monitoring/metrics-server-6ff6c7b584-hbbsv" Sep 12 01:46:21.487371 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:21.487328 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/01af4e11-0c36-4ddb-afa6-79acbc8ec2c1-secret-metrics-server-client-certs\") pod \"metrics-server-6ff6c7b584-hbbsv\" (UID: \"01af4e11-0c36-4ddb-afa6-79acbc8ec2c1\") " pod="openshift-monitoring/metrics-server-6ff6c7b584-hbbsv" Sep 12 01:46:21.487548 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:21.487385 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/01af4e11-0c36-4ddb-afa6-79acbc8ec2c1-metrics-server-audit-profiles\") pod \"metrics-server-6ff6c7b584-hbbsv\" (UID: \"01af4e11-0c36-4ddb-afa6-79acbc8ec2c1\") " pod="openshift-monitoring/metrics-server-6ff6c7b584-hbbsv" Sep 12 01:46:21.487548 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:21.487436 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/01af4e11-0c36-4ddb-afa6-79acbc8ec2c1-secret-metrics-server-tls\") pod \"metrics-server-6ff6c7b584-hbbsv\" (UID: \"01af4e11-0c36-4ddb-afa6-79acbc8ec2c1\") " pod="openshift-monitoring/metrics-server-6ff6c7b584-hbbsv" Sep 12 01:46:21.487548 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:21.487485 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzt6s\" (UniqueName: \"kubernetes.io/projected/01af4e11-0c36-4ddb-afa6-79acbc8ec2c1-kube-api-access-kzt6s\") pod \"metrics-server-6ff6c7b584-hbbsv\" (UID: \"01af4e11-0c36-4ddb-afa6-79acbc8ec2c1\") " pod="openshift-monitoring/metrics-server-6ff6c7b584-hbbsv" Sep 12 01:46:21.487548 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:21.487528 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/01af4e11-0c36-4ddb-afa6-79acbc8ec2c1-audit-log\") pod \"metrics-server-6ff6c7b584-hbbsv\" (UID: \"01af4e11-0c36-4ddb-afa6-79acbc8ec2c1\") " pod="openshift-monitoring/metrics-server-6ff6c7b584-hbbsv" Sep 12 01:46:21.487747 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:21.487554 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/01af4e11-0c36-4ddb-afa6-79acbc8ec2c1-client-ca-bundle\") pod \"metrics-server-6ff6c7b584-hbbsv\" (UID: \"01af4e11-0c36-4ddb-afa6-79acbc8ec2c1\") " pod="openshift-monitoring/metrics-server-6ff6c7b584-hbbsv" Sep 12 01:46:21.487747 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:21.487593 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01af4e11-0c36-4ddb-afa6-79acbc8ec2c1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6ff6c7b584-hbbsv\" (UID: \"01af4e11-0c36-4ddb-afa6-79acbc8ec2c1\") " pod="openshift-monitoring/metrics-server-6ff6c7b584-hbbsv" Sep 12 01:46:21.488318 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:21.487929 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/01af4e11-0c36-4ddb-afa6-79acbc8ec2c1-audit-log\") pod \"metrics-server-6ff6c7b584-hbbsv\" (UID: \"01af4e11-0c36-4ddb-afa6-79acbc8ec2c1\") " pod="openshift-monitoring/metrics-server-6ff6c7b584-hbbsv" Sep 12 01:46:21.488433 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:21.488318 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01af4e11-0c36-4ddb-afa6-79acbc8ec2c1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6ff6c7b584-hbbsv\" (UID: \"01af4e11-0c36-4ddb-afa6-79acbc8ec2c1\") " pod="openshift-monitoring/metrics-server-6ff6c7b584-hbbsv" Sep 12 01:46:21.488888 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:21.488862 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/01af4e11-0c36-4ddb-afa6-79acbc8ec2c1-metrics-server-audit-profiles\") pod \"metrics-server-6ff6c7b584-hbbsv\" (UID: \"01af4e11-0c36-4ddb-afa6-79acbc8ec2c1\") " pod="openshift-monitoring/metrics-server-6ff6c7b584-hbbsv" Sep 12 01:46:21.490379 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:21.490356 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/01af4e11-0c36-4ddb-afa6-79acbc8ec2c1-secret-metrics-server-tls\") pod \"metrics-server-6ff6c7b584-hbbsv\" (UID: \"01af4e11-0c36-4ddb-afa6-79acbc8ec2c1\") " pod="openshift-monitoring/metrics-server-6ff6c7b584-hbbsv" Sep 12 01:46:21.490603 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:21.490580 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/01af4e11-0c36-4ddb-afa6-79acbc8ec2c1-secret-metrics-server-client-certs\") pod \"metrics-server-6ff6c7b584-hbbsv\" (UID: \"01af4e11-0c36-4ddb-afa6-79acbc8ec2c1\") " pod="openshift-monitoring/metrics-server-6ff6c7b584-hbbsv" Sep 12 01:46:21.490698 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:21.490642 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/01af4e11-0c36-4ddb-afa6-79acbc8ec2c1-client-ca-bundle\") pod \"metrics-server-6ff6c7b584-hbbsv\" (UID: \"01af4e11-0c36-4ddb-afa6-79acbc8ec2c1\") " pod="openshift-monitoring/metrics-server-6ff6c7b584-hbbsv" Sep 12 01:46:21.500473 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:21.500422 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzt6s\" (UniqueName: \"kubernetes.io/projected/01af4e11-0c36-4ddb-afa6-79acbc8ec2c1-kube-api-access-kzt6s\") pod \"metrics-server-6ff6c7b584-hbbsv\" (UID: \"01af4e11-0c36-4ddb-afa6-79acbc8ec2c1\") " pod="openshift-monitoring/metrics-server-6ff6c7b584-hbbsv" Sep 12 01:46:21.543916 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:21.543880 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6ff6c7b584-hbbsv" Sep 12 01:46:21.702170 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:21.702089 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-bnt6v"] Sep 12 01:46:21.704726 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:21.704700 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-bnt6v" Sep 12 01:46:21.706023 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:21.706000 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:21.707080 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:21.707060 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-lhzqg\"" Sep 12 01:46:21.707257 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:21.707239 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 12 01:46:21.725866 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:21.725834 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-bnt6v"] Sep 12 01:46:21.760371 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:21.760334 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:21.790375 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:21.790337 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/81ea416f-0dd9-46dc-8c78-ab1a4360ec5f-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-bnt6v\" (UID: \"81ea416f-0dd9-46dc-8c78-ab1a4360ec5f\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-bnt6v" Sep 12 01:46:21.790564 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:21.790421 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcfbf\" (UniqueName: \"kubernetes.io/projected/81ea416f-0dd9-46dc-8c78-ab1a4360ec5f-kube-api-access-mcfbf\") pod \"monitoring-plugin-5d85778b9d-bnt6v\" (UID: \"81ea416f-0dd9-46dc-8c78-ab1a4360ec5f\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-bnt6v" Sep 12 01:46:21.891697 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:21.891659 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/81ea416f-0dd9-46dc-8c78-ab1a4360ec5f-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-bnt6v\" (UID: \"81ea416f-0dd9-46dc-8c78-ab1a4360ec5f\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-bnt6v" Sep 12 01:46:21.891882 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:21.891757 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcfbf\" (UniqueName: \"kubernetes.io/projected/81ea416f-0dd9-46dc-8c78-ab1a4360ec5f-kube-api-access-mcfbf\") pod \"monitoring-plugin-5d85778b9d-bnt6v\" (UID: \"81ea416f-0dd9-46dc-8c78-ab1a4360ec5f\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-bnt6v" Sep 12 01:46:21.891882 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:46:21.891814 2579 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Sep 12 01:46:21.891998 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:46:21.891896 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81ea416f-0dd9-46dc-8c78-ab1a4360ec5f-monitoring-plugin-cert podName:81ea416f-0dd9-46dc-8c78-ab1a4360ec5f nodeName:}" failed. No retries permitted until 2026-09-12 01:46:22.391876161 +0000 UTC m=+85.183869934 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/81ea416f-0dd9-46dc-8c78-ab1a4360ec5f-monitoring-plugin-cert") pod "monitoring-plugin-5d85778b9d-bnt6v" (UID: "81ea416f-0dd9-46dc-8c78-ab1a4360ec5f") : secret "monitoring-plugin-cert" not found Sep 12 01:46:21.900962 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:21.900931 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcfbf\" (UniqueName: \"kubernetes.io/projected/81ea416f-0dd9-46dc-8c78-ab1a4360ec5f-kube-api-access-mcfbf\") pod \"monitoring-plugin-5d85778b9d-bnt6v\" (UID: \"81ea416f-0dd9-46dc-8c78-ab1a4360ec5f\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-bnt6v" Sep 12 01:46:22.297957 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:22.297921 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:22.396932 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:22.396895 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/81ea416f-0dd9-46dc-8c78-ab1a4360ec5f-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-bnt6v\" (UID: \"81ea416f-0dd9-46dc-8c78-ab1a4360ec5f\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-bnt6v" Sep 12 01:46:22.399903 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:22.399871 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/81ea416f-0dd9-46dc-8c78-ab1a4360ec5f-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-bnt6v\" (UID: \"81ea416f-0dd9-46dc-8c78-ab1a4360ec5f\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-bnt6v" Sep 12 01:46:22.553689 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:22.553589 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:22.616314 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:22.616282 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-bnt6v" Sep 12 01:46:22.654609 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:22.654574 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:23.128676 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.128645 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:23.134689 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.134657 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 12 01:46:23.142521 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.142490 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:46:23.148377 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.148350 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 12 01:46:23.148534 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.148375 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 12 01:46:23.148534 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.148523 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 12 01:46:23.148721 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.148434 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 12 01:46:23.148922 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.148899 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 12 01:46:23.148979 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.148954 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 12 01:46:23.149348 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.149326 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 12 01:46:23.149587 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.149569 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6bct4v63ttmfp\"" Sep 12 01:46:23.149748 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.149730 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 12 01:46:23.149893 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.149878 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 12 01:46:23.150001 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.149878 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 12 01:46:23.150063 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.150052 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-q269p\"" Sep 12 01:46:23.150190 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.150173 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 12 01:46:23.150334 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.150318 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 12 01:46:23.152713 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.152573 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 12 01:46:23.154505 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.154481 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 12 01:46:23.163946 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.163918 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 12 01:46:23.307427 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.307392 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/82e2cde4-77ff-4ae8-9161-5518af79e161-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:46:23.307427 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.307434 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/82e2cde4-77ff-4ae8-9161-5518af79e161-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:46:23.307956 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.307475 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/82e2cde4-77ff-4ae8-9161-5518af79e161-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:46:23.307956 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.307497 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgd5l\" (UniqueName: \"kubernetes.io/projected/82e2cde4-77ff-4ae8-9161-5518af79e161-kube-api-access-qgd5l\") pod \"prometheus-k8s-0\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:46:23.307956 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.307534 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/82e2cde4-77ff-4ae8-9161-5518af79e161-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:46:23.307956 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.307567 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/82e2cde4-77ff-4ae8-9161-5518af79e161-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:46:23.307956 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.307595 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/82e2cde4-77ff-4ae8-9161-5518af79e161-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:46:23.307956 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.307620 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82e2cde4-77ff-4ae8-9161-5518af79e161-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:46:23.307956 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.307661 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/82e2cde4-77ff-4ae8-9161-5518af79e161-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:46:23.307956 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.307695 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/82e2cde4-77ff-4ae8-9161-5518af79e161-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:46:23.307956 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.307723 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/82e2cde4-77ff-4ae8-9161-5518af79e161-config-out\") pod \"prometheus-k8s-0\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:46:23.307956 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.307766 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/82e2cde4-77ff-4ae8-9161-5518af79e161-config\") pod \"prometheus-k8s-0\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:46:23.307956 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.307796 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/82e2cde4-77ff-4ae8-9161-5518af79e161-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:46:23.307956 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.307815 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/82e2cde4-77ff-4ae8-9161-5518af79e161-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:46:23.307956 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.307839 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/82e2cde4-77ff-4ae8-9161-5518af79e161-web-config\") pod \"prometheus-k8s-0\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:46:23.307956 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.307894 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/82e2cde4-77ff-4ae8-9161-5518af79e161-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:46:23.307956 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.307930 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82e2cde4-77ff-4ae8-9161-5518af79e161-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:46:23.307956 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.307954 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82e2cde4-77ff-4ae8-9161-5518af79e161-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:46:23.308689 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.307984 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/82e2cde4-77ff-4ae8-9161-5518af79e161-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:46:23.308689 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.308018 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/82e2cde4-77ff-4ae8-9161-5518af79e161-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:46:23.408832 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.408743 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/82e2cde4-77ff-4ae8-9161-5518af79e161-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:46:23.408832 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.408790 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgd5l\" (UniqueName: \"kubernetes.io/projected/82e2cde4-77ff-4ae8-9161-5518af79e161-kube-api-access-qgd5l\") pod \"prometheus-k8s-0\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:46:23.409063 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.408835 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/82e2cde4-77ff-4ae8-9161-5518af79e161-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:46:23.409063 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.408866 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/82e2cde4-77ff-4ae8-9161-5518af79e161-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:46:23.409063 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.408891 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/82e2cde4-77ff-4ae8-9161-5518af79e161-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:46:23.409063 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.408913 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82e2cde4-77ff-4ae8-9161-5518af79e161-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:46:23.409063 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.408947 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/82e2cde4-77ff-4ae8-9161-5518af79e161-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:46:23.409063 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.408976 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/82e2cde4-77ff-4ae8-9161-5518af79e161-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:46:23.409063 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.409001 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/82e2cde4-77ff-4ae8-9161-5518af79e161-config-out\") pod \"prometheus-k8s-0\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:46:23.409063 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.409040 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/82e2cde4-77ff-4ae8-9161-5518af79e161-config\") pod \"prometheus-k8s-0\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:46:23.409411 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.409066 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/82e2cde4-77ff-4ae8-9161-5518af79e161-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:46:23.409411 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.409089 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/82e2cde4-77ff-4ae8-9161-5518af79e161-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:46:23.409411 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.409116 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/82e2cde4-77ff-4ae8-9161-5518af79e161-web-config\") pod \"prometheus-k8s-0\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:46:23.409411 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.409176 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/82e2cde4-77ff-4ae8-9161-5518af79e161-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:46:23.409411 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.409203 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82e2cde4-77ff-4ae8-9161-5518af79e161-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:46:23.409411 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.409227 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82e2cde4-77ff-4ae8-9161-5518af79e161-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:46:23.409411 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.409257 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/82e2cde4-77ff-4ae8-9161-5518af79e161-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:46:23.409411 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.409293 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/82e2cde4-77ff-4ae8-9161-5518af79e161-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:46:23.409411 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.409320 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/82e2cde4-77ff-4ae8-9161-5518af79e161-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:46:23.409411 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.409344 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/82e2cde4-77ff-4ae8-9161-5518af79e161-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:46:23.411160 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.411128 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82e2cde4-77ff-4ae8-9161-5518af79e161-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:46:23.411336 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.411311 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/82e2cde4-77ff-4ae8-9161-5518af79e161-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:46:23.411411 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.411366 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82e2cde4-77ff-4ae8-9161-5518af79e161-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:46:23.412199 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.412168 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82e2cde4-77ff-4ae8-9161-5518af79e161-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:46:23.413572 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.413544 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/82e2cde4-77ff-4ae8-9161-5518af79e161-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:46:23.414932 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.414902 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/82e2cde4-77ff-4ae8-9161-5518af79e161-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:46:23.415549 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.415521 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/82e2cde4-77ff-4ae8-9161-5518af79e161-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:46:23.417043 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.417019 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/82e2cde4-77ff-4ae8-9161-5518af79e161-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:46:23.417128 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.417079 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/82e2cde4-77ff-4ae8-9161-5518af79e161-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:46:23.417834 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.417789 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/82e2cde4-77ff-4ae8-9161-5518af79e161-config-out\") pod \"prometheus-k8s-0\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:46:23.418735 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.418696 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/82e2cde4-77ff-4ae8-9161-5518af79e161-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:46:23.419108 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.419064 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/82e2cde4-77ff-4ae8-9161-5518af79e161-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:46:23.419441 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.419413 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/82e2cde4-77ff-4ae8-9161-5518af79e161-web-config\") pod \"prometheus-k8s-0\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:46:23.420385 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.420339 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/82e2cde4-77ff-4ae8-9161-5518af79e161-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:46:23.421521 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.421492 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/82e2cde4-77ff-4ae8-9161-5518af79e161-config\") pod \"prometheus-k8s-0\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:46:23.421733 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.421709 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgd5l\" (UniqueName: \"kubernetes.io/projected/82e2cde4-77ff-4ae8-9161-5518af79e161-kube-api-access-qgd5l\") pod \"prometheus-k8s-0\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:46:23.422122 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.422105 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/82e2cde4-77ff-4ae8-9161-5518af79e161-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:46:23.422425 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.422389 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/82e2cde4-77ff-4ae8-9161-5518af79e161-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:46:23.422938 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.422909 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/82e2cde4-77ff-4ae8-9161-5518af79e161-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:46:23.423110 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.423087 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/82e2cde4-77ff-4ae8-9161-5518af79e161-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:46:23.460366 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:23.460330 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:46:24.001984 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:24.001954 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7dddc48645-jt7vp" Sep 12 01:46:24.117370 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:24.117333 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:24.215772 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:24.215734 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:25.639855 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:25.635801 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-f67ddbc77-l6kn6"] Sep 12 01:46:25.650550 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:25.650520 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8475968459-hkw8j"] Sep 12 01:46:25.656312 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:46:25.656280 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod511982d8_0ade_4304_8d0d_ccb1f951fcb2.slice/crio-66914fd6ccb07dc25fc32c6bdd318e2867fdc8d53741683ba644039b2d6777bf WatchSource:0}: Error finding container 66914fd6ccb07dc25fc32c6bdd318e2867fdc8d53741683ba644039b2d6777bf: Status 404 returned error can't find the container with id 66914fd6ccb07dc25fc32c6bdd318e2867fdc8d53741683ba644039b2d6777bf Sep 12 01:46:25.675381 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:25.674942 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 12 01:46:25.675381 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:25.675041 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6ff6c7b584-hbbsv"] Sep 12 01:46:25.676113 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:46:25.676089 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82e2cde4_77ff_4ae8_9161_5518af79e161.slice/crio-6fd2fe2a6db48658fc5c05d577a3a07531a261dbb9af1f102c27249f499a5148 WatchSource:0}: Error finding container 6fd2fe2a6db48658fc5c05d577a3a07531a261dbb9af1f102c27249f499a5148: Status 404 returned error can't find the container with id 6fd2fe2a6db48658fc5c05d577a3a07531a261dbb9af1f102c27249f499a5148 Sep 12 01:46:25.679952 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:46:25.679926 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01af4e11_0c36_4ddb_afa6_79acbc8ec2c1.slice/crio-2b29c7182585a195800c91b67e9a338400b8abb3aad6eccbd604dbb60d17ae9d WatchSource:0}: Error finding container 2b29c7182585a195800c91b67e9a338400b8abb3aad6eccbd604dbb60d17ae9d: Status 404 returned error can't find the container with id 2b29c7182585a195800c91b67e9a338400b8abb3aad6eccbd604dbb60d17ae9d Sep 12 01:46:25.719487 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:25.719434 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-bnt6v"] Sep 12 01:46:25.730111 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:25.730080 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 12 01:46:25.733754 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:25.733725 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:25.736236 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:46:25.735835 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc87497e0_9439_45a2_896b_dfa72ea98368.slice/crio-ff45f85b3024b61763345b6e6fc4e2e87bd87459b1d56aff3a058241962bb60d WatchSource:0}: Error finding container ff45f85b3024b61763345b6e6fc4e2e87bd87459b1d56aff3a058241962bb60d: Status 404 returned error can't find the container with id ff45f85b3024b61763345b6e6fc4e2e87bd87459b1d56aff3a058241962bb60d Sep 12 01:46:25.785365 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:25.785302 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:26.119271 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:26.119176 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8475968459-hkw8j" event={"ID":"511982d8-0ade-4304-8d0d-ccb1f951fcb2","Type":"ContainerStarted","Data":"66914fd6ccb07dc25fc32c6bdd318e2867fdc8d53741683ba644039b2d6777bf"} Sep 12 01:46:26.123838 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:26.122674 2579 generic.go:356] "Generic (PLEG): container finished" podID="f8921497-7f53-4874-ba19-3528fe3f7cd3" containerID="8144d53758a3e32f1b9b459a73500fe0074d207c241927d310f49369cad0ce40" exitCode=0 Sep 12 01:46:26.123838 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:26.123517 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8nbjq" event={"ID":"f8921497-7f53-4874-ba19-3528fe3f7cd3","Type":"ContainerDied","Data":"8144d53758a3e32f1b9b459a73500fe0074d207c241927d310f49369cad0ce40"} Sep 12 01:46:26.128314 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:26.125730 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-mgsgx" event={"ID":"9fbbd8af-609e-4eac-927a-0d90c136858b","Type":"ContainerStarted","Data":"61b800ba7ba4102b94d72b1ad86655256021c3dd29ce01143e410b43eefdb769"} Sep 12 01:46:26.129506 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:26.128493 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-mgsgx" Sep 12 01:46:26.131552 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:26.131500 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8475968459-hkw8j"] Sep 12 01:46:26.131743 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:26.131720 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-bnt6v" event={"ID":"81ea416f-0dd9-46dc-8c78-ab1a4360ec5f","Type":"ContainerStarted","Data":"3ebf97a2f9f1d7ceacc2c35eaa4f16b70618438b128ae99f76f3cad35ddbcdcd"} Sep 12 01:46:26.134098 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:26.134028 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6ff6c7b584-hbbsv" event={"ID":"01af4e11-0c36-4ddb-afa6-79acbc8ec2c1","Type":"ContainerStarted","Data":"2b29c7182585a195800c91b67e9a338400b8abb3aad6eccbd604dbb60d17ae9d"} Sep 12 01:46:26.136183 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:26.136156 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c87497e0-9439-45a2-896b-dfa72ea98368","Type":"ContainerStarted","Data":"ff45f85b3024b61763345b6e6fc4e2e87bd87459b1d56aff3a058241962bb60d"} Sep 12 01:46:26.138110 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:26.138045 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-f67ddbc77-l6kn6" event={"ID":"7905e746-9f1b-41cb-af97-43f87dc75594","Type":"ContainerStarted","Data":"d6d4487553e2e7923808cec4c76b202c0c5f623999b82777cf2905ec79a742ab"} Sep 12 01:46:26.141039 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:26.140823 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-mgsgx" Sep 12 01:46:26.141039 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:26.140961 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"82e2cde4-77ff-4ae8-9161-5518af79e161","Type":"ContainerStarted","Data":"6fd2fe2a6db48658fc5c05d577a3a07531a261dbb9af1f102c27249f499a5148"} Sep 12 01:46:27.151091 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:27.151056 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8nbjq" event={"ID":"f8921497-7f53-4874-ba19-3528fe3f7cd3","Type":"ContainerStarted","Data":"62efe4485101925531d881aaaf94225a3b47fb055eb3536bd67cc6f126f88e39"} Sep 12 01:46:27.535099 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:27.535041 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:27.643633 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:27.643583 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:28.062336 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:28.062175 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:28.119943 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:28.119856 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:28.177918 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:28.176909 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8nbjq" event={"ID":"f8921497-7f53-4874-ba19-3528fe3f7cd3","Type":"ContainerStarted","Data":"288f4c87d3ecd23abf63a5c29a01c1b7135288c98ce7c58f37a88c15757ce91b"} Sep 12 01:46:28.183326 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:28.181580 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cwrwp" event={"ID":"41d0d41a-0c50-40f7-b01c-99bdfb61cc18","Type":"ContainerStarted","Data":"7b93d3ac092c7414d1e655192226a3768f2b80905fef5badad61621ff4f7a6cb"} Sep 12 01:46:28.183326 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:28.181672 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cwrwp" event={"ID":"41d0d41a-0c50-40f7-b01c-99bdfb61cc18","Type":"ContainerStarted","Data":"502d0d0038e20699b723e06279d06b14e05020f7c21d20d28bc9f01cf3863b94"} Sep 12 01:46:28.200292 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:28.200214 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-mgsgx" podStartSLOduration=4.11593269 podStartE2EDuration="23.200191466s" totalImagesPullingTime="19.084258776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:46:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:46:26.166625585 +0000 UTC m=+88.958619382" watchObservedRunningTime="2026-09-12 01:46:28.200191466 +0000 UTC m=+90.992185267" Sep 12 01:46:28.201076 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:28.201037 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8nbjq" podStartSLOduration=4.043611429 podStartE2EDuration="12.201019206s" totalImagesPullingTime="8.063845847s" totalInitContainerRuntime="93.56193ms" isStatefulPod=true podCreationTimestamp="2026-09-12 01:46:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:46:28.198694822 +0000 UTC m=+90.990688632" watchObservedRunningTime="2026-09-12 01:46:28.201019206 +0000 UTC m=+90.993012993" Sep 12 01:46:28.332637 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:28.332499 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:28.431340 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:28.431277 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:28.531804 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:28.531742 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:28.634803 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:28.634719 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:31.132355 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:31.132258 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:31.226442 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:31.225986 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:34.222331 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:34.221713 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-bnt6v" event={"ID":"81ea416f-0dd9-46dc-8c78-ab1a4360ec5f","Type":"ContainerStarted","Data":"fc3283e252238073308f97e5aad58fc6e66d9000170984d033ece792be4b079b"} Sep 12 01:46:34.222331 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:34.222285 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-bnt6v" Sep 12 01:46:34.226499 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:34.225895 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6ff6c7b584-hbbsv" event={"ID":"01af4e11-0c36-4ddb-afa6-79acbc8ec2c1","Type":"ContainerStarted","Data":"cf09188fdc8daa353adacd6cffe184b38e791eb604d0bc9e134da11859abd26f"} Sep 12 01:46:34.228853 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:34.228829 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-bnt6v" Sep 12 01:46:34.231167 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:34.230357 2579 generic.go:356] "Generic (PLEG): container finished" podID="c87497e0-9439-45a2-896b-dfa72ea98368" containerID="3f0f381dd59e8fbaa074184e7c7cd86259c40ac1ff0b4a8a7f7c528935bc0490" exitCode=0 Sep 12 01:46:34.231167 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:34.231090 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c87497e0-9439-45a2-896b-dfa72ea98368","Type":"ContainerDied","Data":"3f0f381dd59e8fbaa074184e7c7cd86259c40ac1ff0b4a8a7f7c528935bc0490"} Sep 12 01:46:34.235528 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:34.235503 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-f67ddbc77-l6kn6" event={"ID":"7905e746-9f1b-41cb-af97-43f87dc75594","Type":"ContainerStarted","Data":"478976e310844d53d34fc1f5ce5b206cb7b5740cc8ad35e6bfb6e19d1d454fc7"} Sep 12 01:46:34.235631 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:34.235584 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-f67ddbc77-l6kn6" event={"ID":"7905e746-9f1b-41cb-af97-43f87dc75594","Type":"ContainerStarted","Data":"309725496c034f4f526cd25fab2d31a0a32066c1b1e132df5bffdc6eca24d82e"} Sep 12 01:46:34.235631 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:34.235627 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-f67ddbc77-l6kn6" event={"ID":"7905e746-9f1b-41cb-af97-43f87dc75594","Type":"ContainerStarted","Data":"99040660468c5f3467c34616e9f474f7b445799b5ded5ddce558129711a69408"} Sep 12 01:46:34.237617 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:34.237501 2579 generic.go:356] "Generic (PLEG): container finished" podID="82e2cde4-77ff-4ae8-9161-5518af79e161" containerID="428ac000e6d6d3ae1c1d8d02ac92ab43db09aada7b63dbd9f1243f791b71427e" exitCode=0 Sep 12 01:46:34.237617 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:34.237594 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"82e2cde4-77ff-4ae8-9161-5518af79e161","Type":"ContainerDied","Data":"428ac000e6d6d3ae1c1d8d02ac92ab43db09aada7b63dbd9f1243f791b71427e"} Sep 12 01:46:34.240422 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:34.240160 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-bnt6v" podStartSLOduration=5.73822011 podStartE2EDuration="13.240142567s" totalImagesPullingTime="7.501922457s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:46:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:46:34.2383013 +0000 UTC m=+97.030295096" watchObservedRunningTime="2026-09-12 01:46:34.240142567 +0000 UTC m=+97.032136365" Sep 12 01:46:34.243307 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:34.243173 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cwrwp" event={"ID":"41d0d41a-0c50-40f7-b01c-99bdfb61cc18","Type":"ContainerStarted","Data":"ce191aa06c847470b8271d91f09cdd671c30f45a1efa8f6f4bdf855b6643e5b7"} Sep 12 01:46:34.246969 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:34.246894 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8475968459-hkw8j" event={"ID":"511982d8-0ade-4304-8d0d-ccb1f951fcb2","Type":"ContainerStarted","Data":"7e81bda7698da3457c1b3ba7b9610a3f83d3a0447bcfe0c6ecf2fc3f93b6f5e7"} Sep 12 01:46:34.259595 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:34.259534 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6ff6c7b584-hbbsv" podStartSLOduration=5.705364333 podStartE2EDuration="13.259509799s" totalImagesPullingTime="7.554145466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:46:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:46:34.258599953 +0000 UTC m=+97.050593748" watchObservedRunningTime="2026-09-12 01:46:34.259509799 +0000 UTC m=+97.051503595" Sep 12 01:46:34.351164 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:34.351046 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cwrwp" podStartSLOduration=8.56870861 podStartE2EDuration="18.351024698s" totalImagesPullingTime="9.782316088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:46:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:46:34.350216886 +0000 UTC m=+97.142210680" watchObservedRunningTime="2026-09-12 01:46:34.351024698 +0000 UTC m=+97.143018494" Sep 12 01:46:34.371988 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:34.370200 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8475968459-hkw8j" podStartSLOduration=6.790712768 podStartE2EDuration="14.370180493s" totalImagesPullingTime="7.579467725s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:46:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:46:34.368360225 +0000 UTC m=+97.160354020" watchObservedRunningTime="2026-09-12 01:46:34.370180493 +0000 UTC m=+97.162174287" Sep 12 01:46:35.261715 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:35.261105 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-f67ddbc77-l6kn6" event={"ID":"7905e746-9f1b-41cb-af97-43f87dc75594","Type":"ContainerStarted","Data":"21b0269c88e6ecbb58b862224e8975d11a23c5b9ba81e44636b3a5aae974f80c"} Sep 12 01:46:35.261715 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:35.261198 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-f67ddbc77-l6kn6" event={"ID":"7905e746-9f1b-41cb-af97-43f87dc75594","Type":"ContainerStarted","Data":"9ce92aae04467ccd3b2acfeb339adfde8986060d73115715254ab99c6616cca7"} Sep 12 01:46:36.270741 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:36.270266 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-f67ddbc77-l6kn6" event={"ID":"7905e746-9f1b-41cb-af97-43f87dc75594","Type":"ContainerStarted","Data":"f1f31d3658c1f2730ff996f6a5670598ade60912e23c827478705ef597aac981"} Sep 12 01:46:36.291136 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:36.291082 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-f67ddbc77-l6kn6" podStartSLOduration=8.504856129 podStartE2EDuration="17.291066338s" totalImagesPullingTime="8.786210209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:46:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:46:36.289897973 +0000 UTC m=+99.081891792" watchObservedRunningTime="2026-09-12 01:46:36.291066338 +0000 UTC m=+99.083060155" Sep 12 01:46:36.817037 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:36.816948 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:36.916774 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:36.916745 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:37.018237 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:37.018120 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:37.278691 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:37.278657 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c87497e0-9439-45a2-896b-dfa72ea98368","Type":"ContainerStarted","Data":"2f60ed9f53d882709a191c95ab3e2084e3e10024f090d66d6f8d581391e95530"} Sep 12 01:46:37.279107 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:37.278805 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c87497e0-9439-45a2-896b-dfa72ea98368","Type":"ContainerStarted","Data":"f9aaf5b816eadf22f6b1bd8c78e07b0e0a4fff31efacf51e5470be6ccfb54853"} Sep 12 01:46:37.279107 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:37.279069 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-f67ddbc77-l6kn6" Sep 12 01:46:37.540314 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:37.540230 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:37.640974 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:37.640938 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:38.286294 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:38.286261 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"82e2cde4-77ff-4ae8-9161-5518af79e161","Type":"ContainerStarted","Data":"afcd635af5077b5fdd6e6174bba572ac1207d2c124e2b910d0fd86fc051f0760"} Sep 12 01:46:38.289484 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:38.289434 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c87497e0-9439-45a2-896b-dfa72ea98368","Type":"ContainerStarted","Data":"ceb3dfe4e57af9aa5928a72754508119a35dc83ac19905e0f26dd45e656f291a"} Sep 12 01:46:38.289623 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:38.289521 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c87497e0-9439-45a2-896b-dfa72ea98368","Type":"ContainerStarted","Data":"e4c72ee1af15bac0a37c40103002e3dcb0f2416382f9dd7aa56426bf8222f5ca"} Sep 12 01:46:38.289623 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:38.289554 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c87497e0-9439-45a2-896b-dfa72ea98368","Type":"ContainerStarted","Data":"e4e64baf3f7bc791f5074a6458d6dacd8d8780a504317c10c97ccc90b2e4ef29"} Sep 12 01:46:38.295524 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:38.295505 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-f67ddbc77-l6kn6" Sep 12 01:46:39.300533 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:39.300490 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c87497e0-9439-45a2-896b-dfa72ea98368","Type":"ContainerStarted","Data":"4b0367c4338fb38ec71789bb8dc1e43e35345f5280a296297eeb0e08d39c92b5"} Sep 12 01:46:39.304623 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:39.304590 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"82e2cde4-77ff-4ae8-9161-5518af79e161","Type":"ContainerStarted","Data":"a285bdb864f80206c67ed281cdb9d87bf400e11b36ce75b0cc9137921aeedaa5"} Sep 12 01:46:39.304772 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:39.304680 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"82e2cde4-77ff-4ae8-9161-5518af79e161","Type":"ContainerStarted","Data":"f6d3434f6b051450df348d35f232a833ffa6047775f7f6e9d8926cd4f0b49de3"} Sep 12 01:46:39.304772 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:39.304724 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"82e2cde4-77ff-4ae8-9161-5518af79e161","Type":"ContainerStarted","Data":"36989c40673921b740a77fd9502791e99a53bd2f5043c035845373d2b1a6b9f5"} Sep 12 01:46:39.304772 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:39.304764 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"82e2cde4-77ff-4ae8-9161-5518af79e161","Type":"ContainerStarted","Data":"865594a92f3313da51183d5bbdb676c9f2881df9d541a8782d6101c25d488e73"} Sep 12 01:46:39.304870 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:39.304803 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"82e2cde4-77ff-4ae8-9161-5518af79e161","Type":"ContainerStarted","Data":"8f952b20dab0ce83cb5d2e1c671ad7f0e791033d18d2a64f8e62adc43614939e"} Sep 12 01:46:39.329660 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:39.329581 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=12.638877557 podStartE2EDuration="22.329557436s" totalImagesPullingTime="9.667942544s" totalInitContainerRuntime="22.737335ms" isStatefulPod=false podCreationTimestamp="2026-09-12 01:46:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:46:39.326979138 +0000 UTC m=+102.118972934" watchObservedRunningTime="2026-09-12 01:46:39.329557436 +0000 UTC m=+102.121551234" Sep 12 01:46:39.359474 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:39.358482 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.915319488 podStartE2EDuration="16.358438751s" totalImagesPullingTime="11.417904864s" totalInitContainerRuntime="25.214399ms" isStatefulPod=false podCreationTimestamp="2026-09-12 01:46:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:46:39.354173443 +0000 UTC m=+102.146167250" watchObservedRunningTime="2026-09-12 01:46:39.358438751 +0000 UTC m=+102.150432548" Sep 12 01:46:40.041189 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:40.041160 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-bxnbn" Sep 12 01:46:41.061493 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:41.061445 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8475968459-hkw8j" Sep 12 01:46:41.544902 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:41.544864 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6ff6c7b584-hbbsv" Sep 12 01:46:41.545072 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:41.544949 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6ff6c7b584-hbbsv" Sep 12 01:46:43.461475 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:43.461405 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:46:52.568446 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:52.568400 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:52.667118 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:52.667075 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:46:59.271770 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:59.271711 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-8475968459-hkw8j" podUID="511982d8-0ade-4304-8d0d-ccb1f951fcb2" containerName="console" containerID="cri-o://7e81bda7698da3457c1b3ba7b9610a3f83d3a0447bcfe0c6ecf2fc3f93b6f5e7" gracePeriod=15 Sep 12 01:46:59.396320 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:59.396285 2579 generic.go:356] "Generic (PLEG): container finished" podID="511982d8-0ade-4304-8d0d-ccb1f951fcb2" containerID="7e81bda7698da3457c1b3ba7b9610a3f83d3a0447bcfe0c6ecf2fc3f93b6f5e7" exitCode=0 Sep 12 01:46:59.396434 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:59.396356 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8475968459-hkw8j" event={"ID":"511982d8-0ade-4304-8d0d-ccb1f951fcb2","Type":"ContainerDied","Data":"7e81bda7698da3457c1b3ba7b9610a3f83d3a0447bcfe0c6ecf2fc3f93b6f5e7"} Sep 12 01:46:59.515947 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:59.515923 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8475968459-hkw8j" Sep 12 01:46:59.685271 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:59.685242 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/511982d8-0ade-4304-8d0d-ccb1f951fcb2-console-oauth-config\") pod \"511982d8-0ade-4304-8d0d-ccb1f951fcb2\" (UID: \"511982d8-0ade-4304-8d0d-ccb1f951fcb2\") " Sep 12 01:46:59.685499 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:59.685293 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jqxjz\" (UniqueName: \"kubernetes.io/projected/511982d8-0ade-4304-8d0d-ccb1f951fcb2-kube-api-access-jqxjz\") pod \"511982d8-0ade-4304-8d0d-ccb1f951fcb2\" (UID: \"511982d8-0ade-4304-8d0d-ccb1f951fcb2\") " Sep 12 01:46:59.685499 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:59.685323 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/511982d8-0ade-4304-8d0d-ccb1f951fcb2-service-ca\") pod \"511982d8-0ade-4304-8d0d-ccb1f951fcb2\" (UID: \"511982d8-0ade-4304-8d0d-ccb1f951fcb2\") " Sep 12 01:46:59.685499 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:59.685368 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/511982d8-0ade-4304-8d0d-ccb1f951fcb2-console-config\") pod \"511982d8-0ade-4304-8d0d-ccb1f951fcb2\" (UID: \"511982d8-0ade-4304-8d0d-ccb1f951fcb2\") " Sep 12 01:46:59.685499 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:59.685395 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/511982d8-0ade-4304-8d0d-ccb1f951fcb2-oauth-serving-cert\") pod \"511982d8-0ade-4304-8d0d-ccb1f951fcb2\" (UID: \"511982d8-0ade-4304-8d0d-ccb1f951fcb2\") " Sep 12 01:46:59.685499 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:59.685447 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/511982d8-0ade-4304-8d0d-ccb1f951fcb2-trusted-ca-bundle\") pod \"511982d8-0ade-4304-8d0d-ccb1f951fcb2\" (UID: \"511982d8-0ade-4304-8d0d-ccb1f951fcb2\") " Sep 12 01:46:59.685741 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:59.685536 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/511982d8-0ade-4304-8d0d-ccb1f951fcb2-console-serving-cert\") pod \"511982d8-0ade-4304-8d0d-ccb1f951fcb2\" (UID: \"511982d8-0ade-4304-8d0d-ccb1f951fcb2\") " Sep 12 01:46:59.685955 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:59.685922 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/511982d8-0ade-4304-8d0d-ccb1f951fcb2-console-config" (OuterVolumeSpecName: "console-config") pod "511982d8-0ade-4304-8d0d-ccb1f951fcb2" (UID: "511982d8-0ade-4304-8d0d-ccb1f951fcb2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 12 01:46:59.685955 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:59.685945 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/511982d8-0ade-4304-8d0d-ccb1f951fcb2-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "511982d8-0ade-4304-8d0d-ccb1f951fcb2" (UID: "511982d8-0ade-4304-8d0d-ccb1f951fcb2"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 12 01:46:59.686258 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:59.685926 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/511982d8-0ade-4304-8d0d-ccb1f951fcb2-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "511982d8-0ade-4304-8d0d-ccb1f951fcb2" (UID: "511982d8-0ade-4304-8d0d-ccb1f951fcb2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 12 01:46:59.686258 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:59.686173 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/511982d8-0ade-4304-8d0d-ccb1f951fcb2-service-ca" (OuterVolumeSpecName: "service-ca") pod "511982d8-0ade-4304-8d0d-ccb1f951fcb2" (UID: "511982d8-0ade-4304-8d0d-ccb1f951fcb2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 12 01:46:59.687821 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:59.687792 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/511982d8-0ade-4304-8d0d-ccb1f951fcb2-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "511982d8-0ade-4304-8d0d-ccb1f951fcb2" (UID: "511982d8-0ade-4304-8d0d-ccb1f951fcb2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 12 01:46:59.687924 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:59.687840 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/511982d8-0ade-4304-8d0d-ccb1f951fcb2-kube-api-access-jqxjz" (OuterVolumeSpecName: "kube-api-access-jqxjz") pod "511982d8-0ade-4304-8d0d-ccb1f951fcb2" (UID: "511982d8-0ade-4304-8d0d-ccb1f951fcb2"). InnerVolumeSpecName "kube-api-access-jqxjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 12 01:46:59.687924 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:59.687894 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/511982d8-0ade-4304-8d0d-ccb1f951fcb2-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "511982d8-0ade-4304-8d0d-ccb1f951fcb2" (UID: "511982d8-0ade-4304-8d0d-ccb1f951fcb2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 12 01:46:59.786812 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:59.786772 2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/511982d8-0ade-4304-8d0d-ccb1f951fcb2-console-serving-cert\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 01:46:59.786812 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:59.786804 2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/511982d8-0ade-4304-8d0d-ccb1f951fcb2-console-oauth-config\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 01:46:59.786812 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:59.786817 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqxjz\" (UniqueName: \"kubernetes.io/projected/511982d8-0ade-4304-8d0d-ccb1f951fcb2-kube-api-access-jqxjz\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 01:46:59.787049 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:59.786829 2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/511982d8-0ade-4304-8d0d-ccb1f951fcb2-service-ca\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 01:46:59.787049 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:59.786841 2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/511982d8-0ade-4304-8d0d-ccb1f951fcb2-console-config\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 01:46:59.787049 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:59.786851 2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/511982d8-0ade-4304-8d0d-ccb1f951fcb2-oauth-serving-cert\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 01:46:59.787049 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:46:59.786862 2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/511982d8-0ade-4304-8d0d-ccb1f951fcb2-trusted-ca-bundle\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 01:47:00.401980 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:00.401938 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8475968459-hkw8j" event={"ID":"511982d8-0ade-4304-8d0d-ccb1f951fcb2","Type":"ContainerDied","Data":"66914fd6ccb07dc25fc32c6bdd318e2867fdc8d53741683ba644039b2d6777bf"} Sep 12 01:47:00.402471 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:00.402012 2579 scope.go:117] "RemoveContainer" containerID="7e81bda7698da3457c1b3ba7b9610a3f83d3a0447bcfe0c6ecf2fc3f93b6f5e7" Sep 12 01:47:00.402471 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:00.401949 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8475968459-hkw8j" Sep 12 01:47:00.422141 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:00.422111 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8475968459-hkw8j"] Sep 12 01:47:00.432125 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:00.429674 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8475968459-hkw8j"] Sep 12 01:47:01.550059 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:01.550030 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6ff6c7b584-hbbsv" Sep 12 01:47:01.554012 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:01.553980 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6ff6c7b584-hbbsv" Sep 12 01:47:01.725441 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:01.725404 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="511982d8-0ade-4304-8d0d-ccb1f951fcb2" path="/var/lib/kubelet/pods/511982d8-0ade-4304-8d0d-ccb1f951fcb2/volumes" Sep 12 01:47:11.073278 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:11.073237 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:47:23.461121 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:23.461078 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:23.481285 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:23.481255 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:23.516887 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:23.516859 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:24.786799 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:24.786769 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:47:36.165809 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:36.165767 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:47:36.489747 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:36.489660 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:47:37.284485 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:37.284427 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 12 01:47:37.284962 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:37.284823 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c87497e0-9439-45a2-896b-dfa72ea98368" containerName="alertmanager" containerID="cri-o://f9aaf5b816eadf22f6b1bd8c78e07b0e0a4fff31efacf51e5470be6ccfb54853" gracePeriod=120 Sep 12 01:47:37.285239 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:37.285209 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c87497e0-9439-45a2-896b-dfa72ea98368" containerName="prom-label-proxy" containerID="cri-o://4b0367c4338fb38ec71789bb8dc1e43e35345f5280a296297eeb0e08d39c92b5" gracePeriod=120 Sep 12 01:47:37.285391 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:37.285317 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c87497e0-9439-45a2-896b-dfa72ea98368" containerName="kube-rbac-proxy-metric" containerID="cri-o://ceb3dfe4e57af9aa5928a72754508119a35dc83ac19905e0f26dd45e656f291a" gracePeriod=120 Sep 12 01:47:37.285473 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:37.285388 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c87497e0-9439-45a2-896b-dfa72ea98368" containerName="kube-rbac-proxy" containerID="cri-o://e4c72ee1af15bac0a37c40103002e3dcb0f2416382f9dd7aa56426bf8222f5ca" gracePeriod=120 Sep 12 01:47:37.285532 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:37.285519 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c87497e0-9439-45a2-896b-dfa72ea98368" containerName="config-reloader" containerID="cri-o://2f60ed9f53d882709a191c95ab3e2084e3e10024f090d66d6f8d581391e95530" gracePeriod=120 Sep 12 01:47:37.286160 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:37.285640 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c87497e0-9439-45a2-896b-dfa72ea98368" containerName="kube-rbac-proxy-web" containerID="cri-o://e4e64baf3f7bc791f5074a6458d6dacd8d8780a504317c10c97ccc90b2e4ef29" gracePeriod=120 Sep 12 01:47:37.574520 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:37.574409 2579 generic.go:356] "Generic (PLEG): container finished" podID="c87497e0-9439-45a2-896b-dfa72ea98368" containerID="4b0367c4338fb38ec71789bb8dc1e43e35345f5280a296297eeb0e08d39c92b5" exitCode=0 Sep 12 01:47:37.574520 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:37.574433 2579 generic.go:356] "Generic (PLEG): container finished" podID="c87497e0-9439-45a2-896b-dfa72ea98368" containerID="ceb3dfe4e57af9aa5928a72754508119a35dc83ac19905e0f26dd45e656f291a" exitCode=0 Sep 12 01:47:37.574520 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:37.574441 2579 generic.go:356] "Generic (PLEG): container finished" podID="c87497e0-9439-45a2-896b-dfa72ea98368" containerID="e4c72ee1af15bac0a37c40103002e3dcb0f2416382f9dd7aa56426bf8222f5ca" exitCode=0 Sep 12 01:47:37.574520 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:37.574447 2579 generic.go:356] "Generic (PLEG): container finished" podID="c87497e0-9439-45a2-896b-dfa72ea98368" containerID="2f60ed9f53d882709a191c95ab3e2084e3e10024f090d66d6f8d581391e95530" exitCode=0 Sep 12 01:47:37.574520 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:37.574474 2579 generic.go:356] "Generic (PLEG): container finished" podID="c87497e0-9439-45a2-896b-dfa72ea98368" containerID="f9aaf5b816eadf22f6b1bd8c78e07b0e0a4fff31efacf51e5470be6ccfb54853" exitCode=0 Sep 12 01:47:37.574520 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:37.574492 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c87497e0-9439-45a2-896b-dfa72ea98368","Type":"ContainerDied","Data":"4b0367c4338fb38ec71789bb8dc1e43e35345f5280a296297eeb0e08d39c92b5"} Sep 12 01:47:37.574802 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:37.574564 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c87497e0-9439-45a2-896b-dfa72ea98368","Type":"ContainerDied","Data":"ceb3dfe4e57af9aa5928a72754508119a35dc83ac19905e0f26dd45e656f291a"} Sep 12 01:47:37.574802 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:37.574599 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c87497e0-9439-45a2-896b-dfa72ea98368","Type":"ContainerDied","Data":"e4c72ee1af15bac0a37c40103002e3dcb0f2416382f9dd7aa56426bf8222f5ca"} Sep 12 01:47:37.574802 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:37.574625 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c87497e0-9439-45a2-896b-dfa72ea98368","Type":"ContainerDied","Data":"2f60ed9f53d882709a191c95ab3e2084e3e10024f090d66d6f8d581391e95530"} Sep 12 01:47:37.574802 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:37.574649 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c87497e0-9439-45a2-896b-dfa72ea98368","Type":"ContainerDied","Data":"f9aaf5b816eadf22f6b1bd8c78e07b0e0a4fff31efacf51e5470be6ccfb54853"} Sep 12 01:47:38.577613 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.577582 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:47:38.582694 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.582657 2579 generic.go:356] "Generic (PLEG): container finished" podID="c87497e0-9439-45a2-896b-dfa72ea98368" containerID="e4e64baf3f7bc791f5074a6458d6dacd8d8780a504317c10c97ccc90b2e4ef29" exitCode=0 Sep 12 01:47:38.582694 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.582698 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c87497e0-9439-45a2-896b-dfa72ea98368","Type":"ContainerDied","Data":"e4e64baf3f7bc791f5074a6458d6dacd8d8780a504317c10c97ccc90b2e4ef29"} Sep 12 01:47:38.582866 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.582735 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:47:38.582866 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.582757 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c87497e0-9439-45a2-896b-dfa72ea98368","Type":"ContainerDied","Data":"ff45f85b3024b61763345b6e6fc4e2e87bd87459b1d56aff3a058241962bb60d"} Sep 12 01:47:38.582866 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.582796 2579 scope.go:117] "RemoveContainer" containerID="4b0367c4338fb38ec71789bb8dc1e43e35345f5280a296297eeb0e08d39c92b5" Sep 12 01:47:38.590630 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.590612 2579 scope.go:117] "RemoveContainer" containerID="ceb3dfe4e57af9aa5928a72754508119a35dc83ac19905e0f26dd45e656f291a" Sep 12 01:47:38.598471 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.598431 2579 scope.go:117] "RemoveContainer" containerID="e4c72ee1af15bac0a37c40103002e3dcb0f2416382f9dd7aa56426bf8222f5ca" Sep 12 01:47:38.606088 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.606056 2579 scope.go:117] "RemoveContainer" containerID="e4e64baf3f7bc791f5074a6458d6dacd8d8780a504317c10c97ccc90b2e4ef29" Sep 12 01:47:38.613141 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.613116 2579 scope.go:117] "RemoveContainer" containerID="2f60ed9f53d882709a191c95ab3e2084e3e10024f090d66d6f8d581391e95530" Sep 12 01:47:38.621433 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.621414 2579 scope.go:117] "RemoveContainer" containerID="f9aaf5b816eadf22f6b1bd8c78e07b0e0a4fff31efacf51e5470be6ccfb54853" Sep 12 01:47:38.631388 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.631365 2579 scope.go:117] "RemoveContainer" containerID="3f0f381dd59e8fbaa074184e7c7cd86259c40ac1ff0b4a8a7f7c528935bc0490" Sep 12 01:47:38.641332 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.641306 2579 scope.go:117] "RemoveContainer" containerID="4b0367c4338fb38ec71789bb8dc1e43e35345f5280a296297eeb0e08d39c92b5" Sep 12 01:47:38.641756 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:47:38.641729 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b0367c4338fb38ec71789bb8dc1e43e35345f5280a296297eeb0e08d39c92b5\": container with ID starting with 4b0367c4338fb38ec71789bb8dc1e43e35345f5280a296297eeb0e08d39c92b5 not found: ID does not exist" containerID="4b0367c4338fb38ec71789bb8dc1e43e35345f5280a296297eeb0e08d39c92b5" Sep 12 01:47:38.641841 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.641770 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b0367c4338fb38ec71789bb8dc1e43e35345f5280a296297eeb0e08d39c92b5"} err="failed to get container status \"4b0367c4338fb38ec71789bb8dc1e43e35345f5280a296297eeb0e08d39c92b5\": rpc error: code = NotFound desc = could not find container \"4b0367c4338fb38ec71789bb8dc1e43e35345f5280a296297eeb0e08d39c92b5\": container with ID starting with 4b0367c4338fb38ec71789bb8dc1e43e35345f5280a296297eeb0e08d39c92b5 not found: ID does not exist" Sep 12 01:47:38.641841 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.641822 2579 scope.go:117] "RemoveContainer" containerID="ceb3dfe4e57af9aa5928a72754508119a35dc83ac19905e0f26dd45e656f291a" Sep 12 01:47:38.642180 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:47:38.642152 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ceb3dfe4e57af9aa5928a72754508119a35dc83ac19905e0f26dd45e656f291a\": container with ID starting with ceb3dfe4e57af9aa5928a72754508119a35dc83ac19905e0f26dd45e656f291a not found: ID does not exist" containerID="ceb3dfe4e57af9aa5928a72754508119a35dc83ac19905e0f26dd45e656f291a" Sep 12 01:47:38.642292 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.642189 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ceb3dfe4e57af9aa5928a72754508119a35dc83ac19905e0f26dd45e656f291a"} err="failed to get container status \"ceb3dfe4e57af9aa5928a72754508119a35dc83ac19905e0f26dd45e656f291a\": rpc error: code = NotFound desc = could not find container \"ceb3dfe4e57af9aa5928a72754508119a35dc83ac19905e0f26dd45e656f291a\": container with ID starting with ceb3dfe4e57af9aa5928a72754508119a35dc83ac19905e0f26dd45e656f291a not found: ID does not exist" Sep 12 01:47:38.642292 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.642213 2579 scope.go:117] "RemoveContainer" containerID="e4c72ee1af15bac0a37c40103002e3dcb0f2416382f9dd7aa56426bf8222f5ca" Sep 12 01:47:38.642619 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:47:38.642597 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4c72ee1af15bac0a37c40103002e3dcb0f2416382f9dd7aa56426bf8222f5ca\": container with ID starting with e4c72ee1af15bac0a37c40103002e3dcb0f2416382f9dd7aa56426bf8222f5ca not found: ID does not exist" containerID="e4c72ee1af15bac0a37c40103002e3dcb0f2416382f9dd7aa56426bf8222f5ca" Sep 12 01:47:38.642726 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.642630 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4c72ee1af15bac0a37c40103002e3dcb0f2416382f9dd7aa56426bf8222f5ca"} err="failed to get container status \"e4c72ee1af15bac0a37c40103002e3dcb0f2416382f9dd7aa56426bf8222f5ca\": rpc error: code = NotFound desc = could not find container \"e4c72ee1af15bac0a37c40103002e3dcb0f2416382f9dd7aa56426bf8222f5ca\": container with ID starting with e4c72ee1af15bac0a37c40103002e3dcb0f2416382f9dd7aa56426bf8222f5ca not found: ID does not exist" Sep 12 01:47:38.642726 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.642650 2579 scope.go:117] "RemoveContainer" containerID="e4e64baf3f7bc791f5074a6458d6dacd8d8780a504317c10c97ccc90b2e4ef29" Sep 12 01:47:38.643151 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:47:38.643132 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4e64baf3f7bc791f5074a6458d6dacd8d8780a504317c10c97ccc90b2e4ef29\": container with ID starting with e4e64baf3f7bc791f5074a6458d6dacd8d8780a504317c10c97ccc90b2e4ef29 not found: ID does not exist" containerID="e4e64baf3f7bc791f5074a6458d6dacd8d8780a504317c10c97ccc90b2e4ef29" Sep 12 01:47:38.643151 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.643152 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4e64baf3f7bc791f5074a6458d6dacd8d8780a504317c10c97ccc90b2e4ef29"} err="failed to get container status \"e4e64baf3f7bc791f5074a6458d6dacd8d8780a504317c10c97ccc90b2e4ef29\": rpc error: code = NotFound desc = could not find container \"e4e64baf3f7bc791f5074a6458d6dacd8d8780a504317c10c97ccc90b2e4ef29\": container with ID starting with e4e64baf3f7bc791f5074a6458d6dacd8d8780a504317c10c97ccc90b2e4ef29 not found: ID does not exist" Sep 12 01:47:38.643329 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.643168 2579 scope.go:117] "RemoveContainer" containerID="2f60ed9f53d882709a191c95ab3e2084e3e10024f090d66d6f8d581391e95530" Sep 12 01:47:38.643668 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:47:38.643435 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f60ed9f53d882709a191c95ab3e2084e3e10024f090d66d6f8d581391e95530\": container with ID starting with 2f60ed9f53d882709a191c95ab3e2084e3e10024f090d66d6f8d581391e95530 not found: ID does not exist" containerID="2f60ed9f53d882709a191c95ab3e2084e3e10024f090d66d6f8d581391e95530" Sep 12 01:47:38.643668 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.643518 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f60ed9f53d882709a191c95ab3e2084e3e10024f090d66d6f8d581391e95530"} err="failed to get container status \"2f60ed9f53d882709a191c95ab3e2084e3e10024f090d66d6f8d581391e95530\": rpc error: code = NotFound desc = could not find container \"2f60ed9f53d882709a191c95ab3e2084e3e10024f090d66d6f8d581391e95530\": container with ID starting with 2f60ed9f53d882709a191c95ab3e2084e3e10024f090d66d6f8d581391e95530 not found: ID does not exist" Sep 12 01:47:38.643668 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.643541 2579 scope.go:117] "RemoveContainer" containerID="f9aaf5b816eadf22f6b1bd8c78e07b0e0a4fff31efacf51e5470be6ccfb54853" Sep 12 01:47:38.644180 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:47:38.644158 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9aaf5b816eadf22f6b1bd8c78e07b0e0a4fff31efacf51e5470be6ccfb54853\": container with ID starting with f9aaf5b816eadf22f6b1bd8c78e07b0e0a4fff31efacf51e5470be6ccfb54853 not found: ID does not exist" containerID="f9aaf5b816eadf22f6b1bd8c78e07b0e0a4fff31efacf51e5470be6ccfb54853" Sep 12 01:47:38.644270 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.644200 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9aaf5b816eadf22f6b1bd8c78e07b0e0a4fff31efacf51e5470be6ccfb54853"} err="failed to get container status \"f9aaf5b816eadf22f6b1bd8c78e07b0e0a4fff31efacf51e5470be6ccfb54853\": rpc error: code = NotFound desc = could not find container \"f9aaf5b816eadf22f6b1bd8c78e07b0e0a4fff31efacf51e5470be6ccfb54853\": container with ID starting with f9aaf5b816eadf22f6b1bd8c78e07b0e0a4fff31efacf51e5470be6ccfb54853 not found: ID does not exist" Sep 12 01:47:38.644270 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.644221 2579 scope.go:117] "RemoveContainer" containerID="3f0f381dd59e8fbaa074184e7c7cd86259c40ac1ff0b4a8a7f7c528935bc0490" Sep 12 01:47:38.644612 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:47:38.644589 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f0f381dd59e8fbaa074184e7c7cd86259c40ac1ff0b4a8a7f7c528935bc0490\": container with ID starting with 3f0f381dd59e8fbaa074184e7c7cd86259c40ac1ff0b4a8a7f7c528935bc0490 not found: ID does not exist" containerID="3f0f381dd59e8fbaa074184e7c7cd86259c40ac1ff0b4a8a7f7c528935bc0490" Sep 12 01:47:38.644681 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.644623 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f0f381dd59e8fbaa074184e7c7cd86259c40ac1ff0b4a8a7f7c528935bc0490"} err="failed to get container status \"3f0f381dd59e8fbaa074184e7c7cd86259c40ac1ff0b4a8a7f7c528935bc0490\": rpc error: code = NotFound desc = could not find container \"3f0f381dd59e8fbaa074184e7c7cd86259c40ac1ff0b4a8a7f7c528935bc0490\": container with ID starting with 3f0f381dd59e8fbaa074184e7c7cd86259c40ac1ff0b4a8a7f7c528935bc0490 not found: ID does not exist" Sep 12 01:47:38.703633 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.703604 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r8286\" (UniqueName: \"kubernetes.io/projected/c87497e0-9439-45a2-896b-dfa72ea98368-kube-api-access-r8286\") pod \"c87497e0-9439-45a2-896b-dfa72ea98368\" (UID: \"c87497e0-9439-45a2-896b-dfa72ea98368\") " Sep 12 01:47:38.703802 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.703646 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c87497e0-9439-45a2-896b-dfa72ea98368-secret-alertmanager-kube-rbac-proxy\") pod \"c87497e0-9439-45a2-896b-dfa72ea98368\" (UID: \"c87497e0-9439-45a2-896b-dfa72ea98368\") " Sep 12 01:47:38.703802 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.703681 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c87497e0-9439-45a2-896b-dfa72ea98368-cluster-tls-config\") pod \"c87497e0-9439-45a2-896b-dfa72ea98368\" (UID: \"c87497e0-9439-45a2-896b-dfa72ea98368\") " Sep 12 01:47:38.703802 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.703705 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c87497e0-9439-45a2-896b-dfa72ea98368-config-out\") pod \"c87497e0-9439-45a2-896b-dfa72ea98368\" (UID: \"c87497e0-9439-45a2-896b-dfa72ea98368\") " Sep 12 01:47:38.703802 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.703751 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c87497e0-9439-45a2-896b-dfa72ea98368-secret-alertmanager-kube-rbac-proxy-metric\") pod \"c87497e0-9439-45a2-896b-dfa72ea98368\" (UID: \"c87497e0-9439-45a2-896b-dfa72ea98368\") " Sep 12 01:47:38.703802 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.703779 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c87497e0-9439-45a2-896b-dfa72ea98368-config-volume\") pod \"c87497e0-9439-45a2-896b-dfa72ea98368\" (UID: \"c87497e0-9439-45a2-896b-dfa72ea98368\") " Sep 12 01:47:38.704052 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.703808 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c87497e0-9439-45a2-896b-dfa72ea98368-alertmanager-trusted-ca-bundle\") pod \"c87497e0-9439-45a2-896b-dfa72ea98368\" (UID: \"c87497e0-9439-45a2-896b-dfa72ea98368\") " Sep 12 01:47:38.704052 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.703841 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c87497e0-9439-45a2-896b-dfa72ea98368-web-config\") pod \"c87497e0-9439-45a2-896b-dfa72ea98368\" (UID: \"c87497e0-9439-45a2-896b-dfa72ea98368\") " Sep 12 01:47:38.704403 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.704319 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c87497e0-9439-45a2-896b-dfa72ea98368-tls-assets\") pod \"c87497e0-9439-45a2-896b-dfa72ea98368\" (UID: \"c87497e0-9439-45a2-896b-dfa72ea98368\") " Sep 12 01:47:38.704624 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.704606 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c87497e0-9439-45a2-896b-dfa72ea98368-alertmanager-main-db\") pod \"c87497e0-9439-45a2-896b-dfa72ea98368\" (UID: \"c87497e0-9439-45a2-896b-dfa72ea98368\") " Sep 12 01:47:38.704703 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.704624 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c87497e0-9439-45a2-896b-dfa72ea98368-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "c87497e0-9439-45a2-896b-dfa72ea98368" (UID: "c87497e0-9439-45a2-896b-dfa72ea98368"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 12 01:47:38.704703 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.704637 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c87497e0-9439-45a2-896b-dfa72ea98368-metrics-client-ca\") pod \"c87497e0-9439-45a2-896b-dfa72ea98368\" (UID: \"c87497e0-9439-45a2-896b-dfa72ea98368\") " Sep 12 01:47:38.704806 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.704705 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c87497e0-9439-45a2-896b-dfa72ea98368-secret-alertmanager-kube-rbac-proxy-web\") pod \"c87497e0-9439-45a2-896b-dfa72ea98368\" (UID: \"c87497e0-9439-45a2-896b-dfa72ea98368\") " Sep 12 01:47:38.704806 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.704737 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c87497e0-9439-45a2-896b-dfa72ea98368-secret-alertmanager-main-tls\") pod \"c87497e0-9439-45a2-896b-dfa72ea98368\" (UID: \"c87497e0-9439-45a2-896b-dfa72ea98368\") " Sep 12 01:47:38.705128 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.705101 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c87497e0-9439-45a2-896b-dfa72ea98368-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "c87497e0-9439-45a2-896b-dfa72ea98368" (UID: "c87497e0-9439-45a2-896b-dfa72ea98368"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 12 01:47:38.705271 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.705254 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c87497e0-9439-45a2-896b-dfa72ea98368-metrics-client-ca\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 01:47:38.705335 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.705274 2579 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c87497e0-9439-45a2-896b-dfa72ea98368-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 01:47:38.705614 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.705587 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c87497e0-9439-45a2-896b-dfa72ea98368-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "c87497e0-9439-45a2-896b-dfa72ea98368" (UID: "c87497e0-9439-45a2-896b-dfa72ea98368"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 12 01:47:38.707189 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.707161 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c87497e0-9439-45a2-896b-dfa72ea98368-config-out" (OuterVolumeSpecName: "config-out") pod "c87497e0-9439-45a2-896b-dfa72ea98368" (UID: "c87497e0-9439-45a2-896b-dfa72ea98368"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 12 01:47:38.707354 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.707321 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c87497e0-9439-45a2-896b-dfa72ea98368-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "c87497e0-9439-45a2-896b-dfa72ea98368" (UID: "c87497e0-9439-45a2-896b-dfa72ea98368"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 12 01:47:38.707439 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.707403 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c87497e0-9439-45a2-896b-dfa72ea98368-kube-api-access-r8286" (OuterVolumeSpecName: "kube-api-access-r8286") pod "c87497e0-9439-45a2-896b-dfa72ea98368" (UID: "c87497e0-9439-45a2-896b-dfa72ea98368"). InnerVolumeSpecName "kube-api-access-r8286". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 12 01:47:38.707776 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.707733 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c87497e0-9439-45a2-896b-dfa72ea98368-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "c87497e0-9439-45a2-896b-dfa72ea98368" (UID: "c87497e0-9439-45a2-896b-dfa72ea98368"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 12 01:47:38.708126 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.708049 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c87497e0-9439-45a2-896b-dfa72ea98368-config-volume" (OuterVolumeSpecName: "config-volume") pod "c87497e0-9439-45a2-896b-dfa72ea98368" (UID: "c87497e0-9439-45a2-896b-dfa72ea98368"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 12 01:47:38.708126 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.708117 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c87497e0-9439-45a2-896b-dfa72ea98368-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "c87497e0-9439-45a2-896b-dfa72ea98368" (UID: "c87497e0-9439-45a2-896b-dfa72ea98368"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 12 01:47:38.708960 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.708930 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c87497e0-9439-45a2-896b-dfa72ea98368-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "c87497e0-9439-45a2-896b-dfa72ea98368" (UID: "c87497e0-9439-45a2-896b-dfa72ea98368"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 12 01:47:38.709420 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.709399 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c87497e0-9439-45a2-896b-dfa72ea98368-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "c87497e0-9439-45a2-896b-dfa72ea98368" (UID: "c87497e0-9439-45a2-896b-dfa72ea98368"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 12 01:47:38.712091 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.711960 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c87497e0-9439-45a2-896b-dfa72ea98368-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "c87497e0-9439-45a2-896b-dfa72ea98368" (UID: "c87497e0-9439-45a2-896b-dfa72ea98368"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 12 01:47:38.719664 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.719621 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c87497e0-9439-45a2-896b-dfa72ea98368-web-config" (OuterVolumeSpecName: "web-config") pod "c87497e0-9439-45a2-896b-dfa72ea98368" (UID: "c87497e0-9439-45a2-896b-dfa72ea98368"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 12 01:47:38.806344 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.806308 2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c87497e0-9439-45a2-896b-dfa72ea98368-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 01:47:38.806344 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.806337 2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c87497e0-9439-45a2-896b-dfa72ea98368-secret-alertmanager-main-tls\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 01:47:38.806344 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.806347 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8286\" (UniqueName: \"kubernetes.io/projected/c87497e0-9439-45a2-896b-dfa72ea98368-kube-api-access-r8286\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 01:47:38.806344 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.806356 2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c87497e0-9439-45a2-896b-dfa72ea98368-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 01:47:38.806639 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.806365 2579 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c87497e0-9439-45a2-896b-dfa72ea98368-cluster-tls-config\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 01:47:38.806639 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.806373 2579 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c87497e0-9439-45a2-896b-dfa72ea98368-config-out\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 01:47:38.806639 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.806383 2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c87497e0-9439-45a2-896b-dfa72ea98368-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 01:47:38.806639 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.806391 2579 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c87497e0-9439-45a2-896b-dfa72ea98368-config-volume\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 01:47:38.806639 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.806400 2579 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c87497e0-9439-45a2-896b-dfa72ea98368-web-config\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 01:47:38.806639 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.806409 2579 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c87497e0-9439-45a2-896b-dfa72ea98368-tls-assets\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 01:47:38.806639 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.806416 2579 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c87497e0-9439-45a2-896b-dfa72ea98368-alertmanager-main-db\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 01:47:38.907077 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.906982 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 12 01:47:38.912832 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.912802 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 12 01:47:38.952478 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.952411 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 12 01:47:38.955928 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.953144 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c87497e0-9439-45a2-896b-dfa72ea98368" containerName="init-config-reloader" Sep 12 01:47:38.955928 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.953169 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c87497e0-9439-45a2-896b-dfa72ea98368" containerName="init-config-reloader" Sep 12 01:47:38.955928 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.953195 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c87497e0-9439-45a2-896b-dfa72ea98368" containerName="prom-label-proxy" Sep 12 01:47:38.955928 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.953204 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c87497e0-9439-45a2-896b-dfa72ea98368" containerName="prom-label-proxy" Sep 12 01:47:38.955928 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.953220 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c87497e0-9439-45a2-896b-dfa72ea98368" containerName="alertmanager" Sep 12 01:47:38.955928 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.953228 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c87497e0-9439-45a2-896b-dfa72ea98368" containerName="alertmanager" Sep 12 01:47:38.955928 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.953244 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c87497e0-9439-45a2-896b-dfa72ea98368" containerName="config-reloader" Sep 12 01:47:38.955928 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.953257 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c87497e0-9439-45a2-896b-dfa72ea98368" containerName="config-reloader" Sep 12 01:47:38.955928 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.953279 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c87497e0-9439-45a2-896b-dfa72ea98368" containerName="kube-rbac-proxy-web" Sep 12 01:47:38.955928 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.953288 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c87497e0-9439-45a2-896b-dfa72ea98368" containerName="kube-rbac-proxy-web" Sep 12 01:47:38.955928 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.953307 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c87497e0-9439-45a2-896b-dfa72ea98368" containerName="kube-rbac-proxy-metric" Sep 12 01:47:38.955928 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.953315 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c87497e0-9439-45a2-896b-dfa72ea98368" containerName="kube-rbac-proxy-metric" Sep 12 01:47:38.955928 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.953334 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="511982d8-0ade-4304-8d0d-ccb1f951fcb2" containerName="console" Sep 12 01:47:38.955928 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.953342 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="511982d8-0ade-4304-8d0d-ccb1f951fcb2" containerName="console" Sep 12 01:47:38.955928 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.953350 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c87497e0-9439-45a2-896b-dfa72ea98368" containerName="kube-rbac-proxy" Sep 12 01:47:38.955928 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.953358 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c87497e0-9439-45a2-896b-dfa72ea98368" containerName="kube-rbac-proxy" Sep 12 01:47:38.955928 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.953525 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c87497e0-9439-45a2-896b-dfa72ea98368" containerName="alertmanager" Sep 12 01:47:38.955928 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.953539 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c87497e0-9439-45a2-896b-dfa72ea98368" containerName="prom-label-proxy" Sep 12 01:47:38.955928 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.953557 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c87497e0-9439-45a2-896b-dfa72ea98368" containerName="kube-rbac-proxy-metric" Sep 12 01:47:38.955928 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.953565 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c87497e0-9439-45a2-896b-dfa72ea98368" containerName="config-reloader" Sep 12 01:47:38.955928 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.953575 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c87497e0-9439-45a2-896b-dfa72ea98368" containerName="kube-rbac-proxy" Sep 12 01:47:38.955928 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.953588 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c87497e0-9439-45a2-896b-dfa72ea98368" containerName="kube-rbac-proxy-web" Sep 12 01:47:38.955928 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.953597 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="511982d8-0ade-4304-8d0d-ccb1f951fcb2" containerName="console" Sep 12 01:47:38.959731 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.959707 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:47:38.965961 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.965891 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 12 01:47:38.966137 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.965960 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 12 01:47:38.966340 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.966155 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 12 01:47:38.966340 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.966288 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-fqfd8\"" Sep 12 01:47:38.966478 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.966382 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 12 01:47:38.966540 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.966531 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 12 01:47:38.966591 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.966550 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 12 01:47:38.966688 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.966667 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 12 01:47:38.966809 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.966784 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 12 01:47:38.971229 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.971199 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 12 01:47:38.982367 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:38.982338 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 12 01:47:39.008105 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:39.008067 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bedf8e26-276b-4209-b34c-77a137548c72-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bedf8e26-276b-4209-b34c-77a137548c72\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:47:39.008278 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:39.008117 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bedf8e26-276b-4209-b34c-77a137548c72-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"bedf8e26-276b-4209-b34c-77a137548c72\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:47:39.008278 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:39.008144 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bedf8e26-276b-4209-b34c-77a137548c72-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"bedf8e26-276b-4209-b34c-77a137548c72\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:47:39.008278 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:39.008186 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bedf8e26-276b-4209-b34c-77a137548c72-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"bedf8e26-276b-4209-b34c-77a137548c72\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:47:39.008278 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:39.008209 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bedf8e26-276b-4209-b34c-77a137548c72-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"bedf8e26-276b-4209-b34c-77a137548c72\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:47:39.008278 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:39.008232 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bedf8e26-276b-4209-b34c-77a137548c72-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"bedf8e26-276b-4209-b34c-77a137548c72\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:47:39.008278 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:39.008252 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv799\" (UniqueName: \"kubernetes.io/projected/bedf8e26-276b-4209-b34c-77a137548c72-kube-api-access-xv799\") pod \"alertmanager-main-0\" (UID: \"bedf8e26-276b-4209-b34c-77a137548c72\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:47:39.008684 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:39.008287 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bedf8e26-276b-4209-b34c-77a137548c72-config-out\") pod \"alertmanager-main-0\" (UID: \"bedf8e26-276b-4209-b34c-77a137548c72\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:47:39.008684 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:39.008329 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bedf8e26-276b-4209-b34c-77a137548c72-web-config\") pod \"alertmanager-main-0\" (UID: \"bedf8e26-276b-4209-b34c-77a137548c72\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:47:39.008684 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:39.008368 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bedf8e26-276b-4209-b34c-77a137548c72-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"bedf8e26-276b-4209-b34c-77a137548c72\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:47:39.008684 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:39.008399 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bedf8e26-276b-4209-b34c-77a137548c72-tls-assets\") pod \"alertmanager-main-0\" (UID: \"bedf8e26-276b-4209-b34c-77a137548c72\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:47:39.008684 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:39.008427 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bedf8e26-276b-4209-b34c-77a137548c72-config-volume\") pod \"alertmanager-main-0\" (UID: \"bedf8e26-276b-4209-b34c-77a137548c72\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:47:39.008684 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:39.008485 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bedf8e26-276b-4209-b34c-77a137548c72-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bedf8e26-276b-4209-b34c-77a137548c72\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:47:39.109694 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:39.109648 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bedf8e26-276b-4209-b34c-77a137548c72-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bedf8e26-276b-4209-b34c-77a137548c72\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:47:39.109694 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:39.109689 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bedf8e26-276b-4209-b34c-77a137548c72-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"bedf8e26-276b-4209-b34c-77a137548c72\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:47:39.109933 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:39.109707 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bedf8e26-276b-4209-b34c-77a137548c72-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"bedf8e26-276b-4209-b34c-77a137548c72\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:47:39.109933 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:39.109830 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bedf8e26-276b-4209-b34c-77a137548c72-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"bedf8e26-276b-4209-b34c-77a137548c72\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:47:39.109933 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:39.109867 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bedf8e26-276b-4209-b34c-77a137548c72-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"bedf8e26-276b-4209-b34c-77a137548c72\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:47:39.109933 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:39.109893 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bedf8e26-276b-4209-b34c-77a137548c72-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"bedf8e26-276b-4209-b34c-77a137548c72\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:47:39.110144 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:39.109937 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv799\" (UniqueName: \"kubernetes.io/projected/bedf8e26-276b-4209-b34c-77a137548c72-kube-api-access-xv799\") pod \"alertmanager-main-0\" (UID: \"bedf8e26-276b-4209-b34c-77a137548c72\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:47:39.110144 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:39.109991 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bedf8e26-276b-4209-b34c-77a137548c72-config-out\") pod \"alertmanager-main-0\" (UID: \"bedf8e26-276b-4209-b34c-77a137548c72\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:47:39.110144 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:39.110034 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bedf8e26-276b-4209-b34c-77a137548c72-web-config\") pod \"alertmanager-main-0\" (UID: \"bedf8e26-276b-4209-b34c-77a137548c72\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:47:39.110144 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:39.110074 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bedf8e26-276b-4209-b34c-77a137548c72-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"bedf8e26-276b-4209-b34c-77a137548c72\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:47:39.110144 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:39.110102 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bedf8e26-276b-4209-b34c-77a137548c72-tls-assets\") pod \"alertmanager-main-0\" (UID: \"bedf8e26-276b-4209-b34c-77a137548c72\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:47:39.110144 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:39.110129 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bedf8e26-276b-4209-b34c-77a137548c72-config-volume\") pod \"alertmanager-main-0\" (UID: \"bedf8e26-276b-4209-b34c-77a137548c72\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:47:39.110411 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:39.110160 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bedf8e26-276b-4209-b34c-77a137548c72-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bedf8e26-276b-4209-b34c-77a137548c72\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:47:39.110411 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:39.110291 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bedf8e26-276b-4209-b34c-77a137548c72-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"bedf8e26-276b-4209-b34c-77a137548c72\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:47:39.111345 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:39.111175 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bedf8e26-276b-4209-b34c-77a137548c72-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"bedf8e26-276b-4209-b34c-77a137548c72\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:47:39.111345 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:39.111298 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bedf8e26-276b-4209-b34c-77a137548c72-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bedf8e26-276b-4209-b34c-77a137548c72\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:47:39.113120 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:39.113068 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bedf8e26-276b-4209-b34c-77a137548c72-config-out\") pod \"alertmanager-main-0\" (UID: \"bedf8e26-276b-4209-b34c-77a137548c72\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:47:39.113219 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:39.113174 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bedf8e26-276b-4209-b34c-77a137548c72-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"bedf8e26-276b-4209-b34c-77a137548c72\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:47:39.113298 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:39.113260 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bedf8e26-276b-4209-b34c-77a137548c72-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"bedf8e26-276b-4209-b34c-77a137548c72\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:47:39.113559 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:39.113524 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bedf8e26-276b-4209-b34c-77a137548c72-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bedf8e26-276b-4209-b34c-77a137548c72\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:47:39.113818 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:39.113802 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bedf8e26-276b-4209-b34c-77a137548c72-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"bedf8e26-276b-4209-b34c-77a137548c72\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:47:39.113914 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:39.113901 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bedf8e26-276b-4209-b34c-77a137548c72-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"bedf8e26-276b-4209-b34c-77a137548c72\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:47:39.114107 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:39.114087 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bedf8e26-276b-4209-b34c-77a137548c72-tls-assets\") pod \"alertmanager-main-0\" (UID: \"bedf8e26-276b-4209-b34c-77a137548c72\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:47:39.114477 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:39.114428 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bedf8e26-276b-4209-b34c-77a137548c72-web-config\") pod \"alertmanager-main-0\" (UID: \"bedf8e26-276b-4209-b34c-77a137548c72\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:47:39.114887 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:39.114870 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bedf8e26-276b-4209-b34c-77a137548c72-config-volume\") pod \"alertmanager-main-0\" (UID: \"bedf8e26-276b-4209-b34c-77a137548c72\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:47:39.119147 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:39.119127 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv799\" (UniqueName: \"kubernetes.io/projected/bedf8e26-276b-4209-b34c-77a137548c72-kube-api-access-xv799\") pod \"alertmanager-main-0\" (UID: \"bedf8e26-276b-4209-b34c-77a137548c72\") " pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:47:39.269710 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:39.269624 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 12 01:47:39.554047 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:39.554024 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 12 01:47:39.556093 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:47:39.556061 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbedf8e26_276b_4209_b34c_77a137548c72.slice/crio-71b6436ad2c4d80e0d130576c6bea8b3e764903ed52c2b159038bfb0963b9ffa WatchSource:0}: Error finding container 71b6436ad2c4d80e0d130576c6bea8b3e764903ed52c2b159038bfb0963b9ffa: Status 404 returned error can't find the container with id 71b6436ad2c4d80e0d130576c6bea8b3e764903ed52c2b159038bfb0963b9ffa Sep 12 01:47:39.588534 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:39.588502 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bedf8e26-276b-4209-b34c-77a137548c72","Type":"ContainerStarted","Data":"71b6436ad2c4d80e0d130576c6bea8b3e764903ed52c2b159038bfb0963b9ffa"} Sep 12 01:47:39.725397 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:39.725365 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c87497e0-9439-45a2-896b-dfa72ea98368" path="/var/lib/kubelet/pods/c87497e0-9439-45a2-896b-dfa72ea98368/volumes" Sep 12 01:47:40.595201 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:40.595166 2579 generic.go:356] "Generic (PLEG): container finished" podID="bedf8e26-276b-4209-b34c-77a137548c72" containerID="03658496a85c7947b02aa323d38cfae997a5bbfde0294a1014498cafdb097a5f" exitCode=0 Sep 12 01:47:40.595579 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:40.595226 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bedf8e26-276b-4209-b34c-77a137548c72","Type":"ContainerDied","Data":"03658496a85c7947b02aa323d38cfae997a5bbfde0294a1014498cafdb097a5f"} Sep 12 01:47:41.251613 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.251573 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-675d55d6d-6rkn4"] Sep 12 01:47:41.260630 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.260590 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-675d55d6d-6rkn4" Sep 12 01:47:41.264443 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.264412 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 12 01:47:41.264685 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.264669 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 12 01:47:41.265236 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.265217 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 12 01:47:41.265305 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.265241 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 12 01:47:41.265347 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.265218 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-q7k6h\"" Sep 12 01:47:41.265587 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.265572 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 12 01:47:41.272391 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.272365 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-675d55d6d-6rkn4"] Sep 12 01:47:41.273127 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.273104 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 12 01:47:41.328848 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.328814 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f746c621-3575-496b-88b5-eaf926350ab1-telemeter-trusted-ca-bundle\") pod \"telemeter-client-675d55d6d-6rkn4\" (UID: \"f746c621-3575-496b-88b5-eaf926350ab1\") " pod="openshift-monitoring/telemeter-client-675d55d6d-6rkn4" Sep 12 01:47:41.328848 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.328853 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f746c621-3575-496b-88b5-eaf926350ab1-serving-certs-ca-bundle\") pod \"telemeter-client-675d55d6d-6rkn4\" (UID: \"f746c621-3575-496b-88b5-eaf926350ab1\") " pod="openshift-monitoring/telemeter-client-675d55d6d-6rkn4" Sep 12 01:47:41.329060 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.328889 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f746c621-3575-496b-88b5-eaf926350ab1-secret-telemeter-client\") pod \"telemeter-client-675d55d6d-6rkn4\" (UID: \"f746c621-3575-496b-88b5-eaf926350ab1\") " pod="openshift-monitoring/telemeter-client-675d55d6d-6rkn4" Sep 12 01:47:41.329060 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.328940 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f746c621-3575-496b-88b5-eaf926350ab1-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-675d55d6d-6rkn4\" (UID: \"f746c621-3575-496b-88b5-eaf926350ab1\") " pod="openshift-monitoring/telemeter-client-675d55d6d-6rkn4" Sep 12 01:47:41.329060 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.328969 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f746c621-3575-496b-88b5-eaf926350ab1-federate-client-tls\") pod \"telemeter-client-675d55d6d-6rkn4\" (UID: \"f746c621-3575-496b-88b5-eaf926350ab1\") " pod="openshift-monitoring/telemeter-client-675d55d6d-6rkn4" Sep 12 01:47:41.329060 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.328990 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svwj4\" (UniqueName: \"kubernetes.io/projected/f746c621-3575-496b-88b5-eaf926350ab1-kube-api-access-svwj4\") pod \"telemeter-client-675d55d6d-6rkn4\" (UID: \"f746c621-3575-496b-88b5-eaf926350ab1\") " pod="openshift-monitoring/telemeter-client-675d55d6d-6rkn4" Sep 12 01:47:41.329060 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.329039 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f746c621-3575-496b-88b5-eaf926350ab1-telemeter-client-tls\") pod \"telemeter-client-675d55d6d-6rkn4\" (UID: \"f746c621-3575-496b-88b5-eaf926350ab1\") " pod="openshift-monitoring/telemeter-client-675d55d6d-6rkn4" Sep 12 01:47:41.329229 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.329067 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f746c621-3575-496b-88b5-eaf926350ab1-metrics-client-ca\") pod \"telemeter-client-675d55d6d-6rkn4\" (UID: \"f746c621-3575-496b-88b5-eaf926350ab1\") " pod="openshift-monitoring/telemeter-client-675d55d6d-6rkn4" Sep 12 01:47:41.430281 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.430234 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f746c621-3575-496b-88b5-eaf926350ab1-secret-telemeter-client\") pod \"telemeter-client-675d55d6d-6rkn4\" (UID: \"f746c621-3575-496b-88b5-eaf926350ab1\") " pod="openshift-monitoring/telemeter-client-675d55d6d-6rkn4" Sep 12 01:47:41.430515 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.430287 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f746c621-3575-496b-88b5-eaf926350ab1-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-675d55d6d-6rkn4\" (UID: \"f746c621-3575-496b-88b5-eaf926350ab1\") " pod="openshift-monitoring/telemeter-client-675d55d6d-6rkn4" Sep 12 01:47:41.430515 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.430317 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f746c621-3575-496b-88b5-eaf926350ab1-federate-client-tls\") pod \"telemeter-client-675d55d6d-6rkn4\" (UID: \"f746c621-3575-496b-88b5-eaf926350ab1\") " pod="openshift-monitoring/telemeter-client-675d55d6d-6rkn4" Sep 12 01:47:41.430515 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.430341 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svwj4\" (UniqueName: \"kubernetes.io/projected/f746c621-3575-496b-88b5-eaf926350ab1-kube-api-access-svwj4\") pod \"telemeter-client-675d55d6d-6rkn4\" (UID: \"f746c621-3575-496b-88b5-eaf926350ab1\") " pod="openshift-monitoring/telemeter-client-675d55d6d-6rkn4" Sep 12 01:47:41.430515 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.430376 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f746c621-3575-496b-88b5-eaf926350ab1-telemeter-client-tls\") pod \"telemeter-client-675d55d6d-6rkn4\" (UID: \"f746c621-3575-496b-88b5-eaf926350ab1\") " pod="openshift-monitoring/telemeter-client-675d55d6d-6rkn4" Sep 12 01:47:41.430515 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.430396 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f746c621-3575-496b-88b5-eaf926350ab1-metrics-client-ca\") pod \"telemeter-client-675d55d6d-6rkn4\" (UID: \"f746c621-3575-496b-88b5-eaf926350ab1\") " pod="openshift-monitoring/telemeter-client-675d55d6d-6rkn4" Sep 12 01:47:41.430515 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.430483 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f746c621-3575-496b-88b5-eaf926350ab1-telemeter-trusted-ca-bundle\") pod \"telemeter-client-675d55d6d-6rkn4\" (UID: \"f746c621-3575-496b-88b5-eaf926350ab1\") " pod="openshift-monitoring/telemeter-client-675d55d6d-6rkn4" Sep 12 01:47:41.430515 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.430510 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f746c621-3575-496b-88b5-eaf926350ab1-serving-certs-ca-bundle\") pod \"telemeter-client-675d55d6d-6rkn4\" (UID: \"f746c621-3575-496b-88b5-eaf926350ab1\") " pod="openshift-monitoring/telemeter-client-675d55d6d-6rkn4" Sep 12 01:47:41.431343 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.431318 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f746c621-3575-496b-88b5-eaf926350ab1-serving-certs-ca-bundle\") pod \"telemeter-client-675d55d6d-6rkn4\" (UID: \"f746c621-3575-496b-88b5-eaf926350ab1\") " pod="openshift-monitoring/telemeter-client-675d55d6d-6rkn4" Sep 12 01:47:41.431691 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.431661 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f746c621-3575-496b-88b5-eaf926350ab1-metrics-client-ca\") pod \"telemeter-client-675d55d6d-6rkn4\" (UID: \"f746c621-3575-496b-88b5-eaf926350ab1\") " pod="openshift-monitoring/telemeter-client-675d55d6d-6rkn4" Sep 12 01:47:41.431795 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.431695 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f746c621-3575-496b-88b5-eaf926350ab1-telemeter-trusted-ca-bundle\") pod \"telemeter-client-675d55d6d-6rkn4\" (UID: \"f746c621-3575-496b-88b5-eaf926350ab1\") " pod="openshift-monitoring/telemeter-client-675d55d6d-6rkn4" Sep 12 01:47:41.433957 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.433609 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f746c621-3575-496b-88b5-eaf926350ab1-secret-telemeter-client\") pod \"telemeter-client-675d55d6d-6rkn4\" (UID: \"f746c621-3575-496b-88b5-eaf926350ab1\") " pod="openshift-monitoring/telemeter-client-675d55d6d-6rkn4" Sep 12 01:47:41.433957 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.433679 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f746c621-3575-496b-88b5-eaf926350ab1-federate-client-tls\") pod \"telemeter-client-675d55d6d-6rkn4\" (UID: \"f746c621-3575-496b-88b5-eaf926350ab1\") " pod="openshift-monitoring/telemeter-client-675d55d6d-6rkn4" Sep 12 01:47:41.434372 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.434347 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f746c621-3575-496b-88b5-eaf926350ab1-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-675d55d6d-6rkn4\" (UID: \"f746c621-3575-496b-88b5-eaf926350ab1\") " pod="openshift-monitoring/telemeter-client-675d55d6d-6rkn4" Sep 12 01:47:41.434944 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.434926 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f746c621-3575-496b-88b5-eaf926350ab1-telemeter-client-tls\") pod \"telemeter-client-675d55d6d-6rkn4\" (UID: \"f746c621-3575-496b-88b5-eaf926350ab1\") " pod="openshift-monitoring/telemeter-client-675d55d6d-6rkn4" Sep 12 01:47:41.439050 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.439019 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svwj4\" (UniqueName: \"kubernetes.io/projected/f746c621-3575-496b-88b5-eaf926350ab1-kube-api-access-svwj4\") pod \"telemeter-client-675d55d6d-6rkn4\" (UID: \"f746c621-3575-496b-88b5-eaf926350ab1\") " pod="openshift-monitoring/telemeter-client-675d55d6d-6rkn4" Sep 12 01:47:41.559919 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.559831 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 12 01:47:41.560508 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.560475 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="82e2cde4-77ff-4ae8-9161-5518af79e161" containerName="prometheus" containerID="cri-o://afcd635af5077b5fdd6e6174bba572ac1207d2c124e2b910d0fd86fc051f0760" gracePeriod=600 Sep 12 01:47:41.561181 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.560663 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="82e2cde4-77ff-4ae8-9161-5518af79e161" containerName="kube-rbac-proxy-thanos" containerID="cri-o://a285bdb864f80206c67ed281cdb9d87bf400e11b36ce75b0cc9137921aeedaa5" gracePeriod=600 Sep 12 01:47:41.561618 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.560896 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="82e2cde4-77ff-4ae8-9161-5518af79e161" containerName="kube-rbac-proxy-web" containerID="cri-o://36989c40673921b740a77fd9502791e99a53bd2f5043c035845373d2b1a6b9f5" gracePeriod=600 Sep 12 01:47:41.561731 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.560916 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="82e2cde4-77ff-4ae8-9161-5518af79e161" containerName="kube-rbac-proxy" containerID="cri-o://f6d3434f6b051450df348d35f232a833ffa6047775f7f6e9d8926cd4f0b49de3" gracePeriod=600 Sep 12 01:47:41.561788 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.560932 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="82e2cde4-77ff-4ae8-9161-5518af79e161" containerName="thanos-sidecar" containerID="cri-o://865594a92f3313da51183d5bbdb676c9f2881df9d541a8782d6101c25d488e73" gracePeriod=600 Sep 12 01:47:41.561854 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.560950 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="82e2cde4-77ff-4ae8-9161-5518af79e161" containerName="config-reloader" containerID="cri-o://8f952b20dab0ce83cb5d2e1c671ad7f0e791033d18d2a64f8e62adc43614939e" gracePeriod=600 Sep 12 01:47:41.574563 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.574292 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-675d55d6d-6rkn4" Sep 12 01:47:41.605586 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.605557 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bedf8e26-276b-4209-b34c-77a137548c72","Type":"ContainerStarted","Data":"120cace1356c7f6a0a65b10e135c9347c35e2923df961e753017b8a4626806a0"} Sep 12 01:47:41.605943 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.605637 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bedf8e26-276b-4209-b34c-77a137548c72","Type":"ContainerStarted","Data":"ba0fc8c96a3bff22b69c3206fa9aef29357c267d23112e73bb2f04051bc36e3b"} Sep 12 01:47:41.605943 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.605680 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bedf8e26-276b-4209-b34c-77a137548c72","Type":"ContainerStarted","Data":"4a3a38d75c144174cb6834a26b11eddd702a0db071de2187381dabec47720263"} Sep 12 01:47:41.605943 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.605721 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bedf8e26-276b-4209-b34c-77a137548c72","Type":"ContainerStarted","Data":"a0547283dc20e5122f639061df9dcdb9435068320851315cd17f4c673cb8df90"} Sep 12 01:47:41.605943 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.605870 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bedf8e26-276b-4209-b34c-77a137548c72","Type":"ContainerStarted","Data":"6641c45249297fdf9ddd3577aa0b17834be6da12ed5897d1cc9f53b0d9722983"} Sep 12 01:47:41.606080 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.605946 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bedf8e26-276b-4209-b34c-77a137548c72","Type":"ContainerStarted","Data":"daf7d29ceb74f8c959d85b0e02f10e4f15f49340ed0baf1a80c92eb40da8c10a"} Sep 12 01:47:41.632150 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.630778 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.613306208 podStartE2EDuration="3.630752837s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.446629ms" isStatefulPod=false podCreationTimestamp="2026-09-12 01:47:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:47:41.630426806 +0000 UTC m=+164.422420636" watchObservedRunningTime="2026-09-12 01:47:41.630752837 +0000 UTC m=+164.422746633" Sep 12 01:47:41.819327 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.819260 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:41.866229 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.866191 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-675d55d6d-6rkn4"] Sep 12 01:47:41.872271 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:47:41.872221 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf746c621_3575_496b_88b5_eaf926350ab1.slice/crio-cd0e4dfb001592a2b9652a5821032c3b1d3d945ba63930efc593064e71e7b1bf WatchSource:0}: Error finding container cd0e4dfb001592a2b9652a5821032c3b1d3d945ba63930efc593064e71e7b1bf: Status 404 returned error can't find the container with id cd0e4dfb001592a2b9652a5821032c3b1d3d945ba63930efc593064e71e7b1bf Sep 12 01:47:41.936681 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.936651 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/82e2cde4-77ff-4ae8-9161-5518af79e161-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"82e2cde4-77ff-4ae8-9161-5518af79e161\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " Sep 12 01:47:41.936681 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.936686 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/82e2cde4-77ff-4ae8-9161-5518af79e161-configmap-metrics-client-ca\") pod \"82e2cde4-77ff-4ae8-9161-5518af79e161\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " Sep 12 01:47:41.936920 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.936710 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/82e2cde4-77ff-4ae8-9161-5518af79e161-prometheus-k8s-rulefiles-2\") pod \"82e2cde4-77ff-4ae8-9161-5518af79e161\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " Sep 12 01:47:41.936920 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.936729 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82e2cde4-77ff-4ae8-9161-5518af79e161-configmap-kubelet-serving-ca-bundle\") pod \"82e2cde4-77ff-4ae8-9161-5518af79e161\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " Sep 12 01:47:41.936920 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.936770 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qgd5l\" (UniqueName: \"kubernetes.io/projected/82e2cde4-77ff-4ae8-9161-5518af79e161-kube-api-access-qgd5l\") pod \"82e2cde4-77ff-4ae8-9161-5518af79e161\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " Sep 12 01:47:41.936920 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.936789 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/82e2cde4-77ff-4ae8-9161-5518af79e161-prometheus-k8s-db\") pod \"82e2cde4-77ff-4ae8-9161-5518af79e161\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " Sep 12 01:47:41.936920 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.936810 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/82e2cde4-77ff-4ae8-9161-5518af79e161-tls-assets\") pod \"82e2cde4-77ff-4ae8-9161-5518af79e161\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " Sep 12 01:47:41.936920 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.936826 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/82e2cde4-77ff-4ae8-9161-5518af79e161-secret-grpc-tls\") pod \"82e2cde4-77ff-4ae8-9161-5518af79e161\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " Sep 12 01:47:41.936920 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.936849 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/82e2cde4-77ff-4ae8-9161-5518af79e161-prometheus-k8s-rulefiles-1\") pod \"82e2cde4-77ff-4ae8-9161-5518af79e161\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " Sep 12 01:47:41.936920 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.936886 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82e2cde4-77ff-4ae8-9161-5518af79e161-configmap-serving-certs-ca-bundle\") pod \"82e2cde4-77ff-4ae8-9161-5518af79e161\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " Sep 12 01:47:41.936920 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.936914 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/82e2cde4-77ff-4ae8-9161-5518af79e161-config-out\") pod \"82e2cde4-77ff-4ae8-9161-5518af79e161\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " Sep 12 01:47:41.937355 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.936947 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/82e2cde4-77ff-4ae8-9161-5518af79e161-web-config\") pod \"82e2cde4-77ff-4ae8-9161-5518af79e161\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " Sep 12 01:47:41.937355 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.936979 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/82e2cde4-77ff-4ae8-9161-5518af79e161-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"82e2cde4-77ff-4ae8-9161-5518af79e161\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " Sep 12 01:47:41.937355 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.937010 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/82e2cde4-77ff-4ae8-9161-5518af79e161-secret-kube-rbac-proxy\") pod \"82e2cde4-77ff-4ae8-9161-5518af79e161\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " Sep 12 01:47:41.937355 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.937035 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/82e2cde4-77ff-4ae8-9161-5518af79e161-secret-metrics-client-certs\") pod \"82e2cde4-77ff-4ae8-9161-5518af79e161\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " Sep 12 01:47:41.937355 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.937033 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82e2cde4-77ff-4ae8-9161-5518af79e161-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "82e2cde4-77ff-4ae8-9161-5518af79e161" (UID: "82e2cde4-77ff-4ae8-9161-5518af79e161"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 12 01:47:41.937355 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.937056 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/82e2cde4-77ff-4ae8-9161-5518af79e161-config\") pod \"82e2cde4-77ff-4ae8-9161-5518af79e161\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " Sep 12 01:47:41.937355 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.937055 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82e2cde4-77ff-4ae8-9161-5518af79e161-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "82e2cde4-77ff-4ae8-9161-5518af79e161" (UID: "82e2cde4-77ff-4ae8-9161-5518af79e161"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 12 01:47:41.937355 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.937138 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82e2cde4-77ff-4ae8-9161-5518af79e161-prometheus-trusted-ca-bundle\") pod \"82e2cde4-77ff-4ae8-9161-5518af79e161\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " Sep 12 01:47:41.937355 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.937170 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/82e2cde4-77ff-4ae8-9161-5518af79e161-prometheus-k8s-rulefiles-0\") pod \"82e2cde4-77ff-4ae8-9161-5518af79e161\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " Sep 12 01:47:41.937355 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.937174 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82e2cde4-77ff-4ae8-9161-5518af79e161-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "82e2cde4-77ff-4ae8-9161-5518af79e161" (UID: "82e2cde4-77ff-4ae8-9161-5518af79e161"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 12 01:47:41.937355 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.937234 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/82e2cde4-77ff-4ae8-9161-5518af79e161-thanos-prometheus-http-client-file\") pod \"82e2cde4-77ff-4ae8-9161-5518af79e161\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " Sep 12 01:47:41.937355 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.937277 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/82e2cde4-77ff-4ae8-9161-5518af79e161-secret-prometheus-k8s-tls\") pod \"82e2cde4-77ff-4ae8-9161-5518af79e161\" (UID: \"82e2cde4-77ff-4ae8-9161-5518af79e161\") " Sep 12 01:47:41.940149 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.937385 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82e2cde4-77ff-4ae8-9161-5518af79e161-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "82e2cde4-77ff-4ae8-9161-5518af79e161" (UID: "82e2cde4-77ff-4ae8-9161-5518af79e161"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 12 01:47:41.940149 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.937662 2579 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/82e2cde4-77ff-4ae8-9161-5518af79e161-configmap-metrics-client-ca\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 01:47:41.940149 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.937683 2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/82e2cde4-77ff-4ae8-9161-5518af79e161-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 01:47:41.940149 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.937699 2579 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82e2cde4-77ff-4ae8-9161-5518af79e161-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 01:47:41.940149 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.937714 2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/82e2cde4-77ff-4ae8-9161-5518af79e161-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 01:47:41.940149 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.937748 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82e2cde4-77ff-4ae8-9161-5518af79e161-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "82e2cde4-77ff-4ae8-9161-5518af79e161" (UID: "82e2cde4-77ff-4ae8-9161-5518af79e161"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 12 01:47:41.940149 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.938021 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82e2cde4-77ff-4ae8-9161-5518af79e161-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "82e2cde4-77ff-4ae8-9161-5518af79e161" (UID: "82e2cde4-77ff-4ae8-9161-5518af79e161"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 12 01:47:41.940149 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.938860 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82e2cde4-77ff-4ae8-9161-5518af79e161-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "82e2cde4-77ff-4ae8-9161-5518af79e161" (UID: "82e2cde4-77ff-4ae8-9161-5518af79e161"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 12 01:47:41.940149 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.939768 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82e2cde4-77ff-4ae8-9161-5518af79e161-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "82e2cde4-77ff-4ae8-9161-5518af79e161" (UID: "82e2cde4-77ff-4ae8-9161-5518af79e161"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 12 01:47:41.940626 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.940377 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82e2cde4-77ff-4ae8-9161-5518af79e161-config-out" (OuterVolumeSpecName: "config-out") pod "82e2cde4-77ff-4ae8-9161-5518af79e161" (UID: "82e2cde4-77ff-4ae8-9161-5518af79e161"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 12 01:47:41.940626 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.940511 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82e2cde4-77ff-4ae8-9161-5518af79e161-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "82e2cde4-77ff-4ae8-9161-5518af79e161" (UID: "82e2cde4-77ff-4ae8-9161-5518af79e161"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 12 01:47:41.941149 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.941111 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82e2cde4-77ff-4ae8-9161-5518af79e161-kube-api-access-qgd5l" (OuterVolumeSpecName: "kube-api-access-qgd5l") pod "82e2cde4-77ff-4ae8-9161-5518af79e161" (UID: "82e2cde4-77ff-4ae8-9161-5518af79e161"). InnerVolumeSpecName "kube-api-access-qgd5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 12 01:47:41.941658 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.941622 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82e2cde4-77ff-4ae8-9161-5518af79e161-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "82e2cde4-77ff-4ae8-9161-5518af79e161" (UID: "82e2cde4-77ff-4ae8-9161-5518af79e161"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 12 01:47:41.941803 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.941783 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82e2cde4-77ff-4ae8-9161-5518af79e161-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "82e2cde4-77ff-4ae8-9161-5518af79e161" (UID: "82e2cde4-77ff-4ae8-9161-5518af79e161"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 12 01:47:41.941863 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.941820 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82e2cde4-77ff-4ae8-9161-5518af79e161-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "82e2cde4-77ff-4ae8-9161-5518af79e161" (UID: "82e2cde4-77ff-4ae8-9161-5518af79e161"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 12 01:47:41.941863 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.941833 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82e2cde4-77ff-4ae8-9161-5518af79e161-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "82e2cde4-77ff-4ae8-9161-5518af79e161" (UID: "82e2cde4-77ff-4ae8-9161-5518af79e161"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 12 01:47:41.942714 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.942693 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82e2cde4-77ff-4ae8-9161-5518af79e161-config" (OuterVolumeSpecName: "config") pod "82e2cde4-77ff-4ae8-9161-5518af79e161" (UID: "82e2cde4-77ff-4ae8-9161-5518af79e161"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 12 01:47:41.942991 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.942974 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82e2cde4-77ff-4ae8-9161-5518af79e161-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "82e2cde4-77ff-4ae8-9161-5518af79e161" (UID: "82e2cde4-77ff-4ae8-9161-5518af79e161"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 12 01:47:41.943179 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.943157 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82e2cde4-77ff-4ae8-9161-5518af79e161-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "82e2cde4-77ff-4ae8-9161-5518af79e161" (UID: "82e2cde4-77ff-4ae8-9161-5518af79e161"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 12 01:47:41.943641 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.943624 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82e2cde4-77ff-4ae8-9161-5518af79e161-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "82e2cde4-77ff-4ae8-9161-5518af79e161" (UID: "82e2cde4-77ff-4ae8-9161-5518af79e161"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 12 01:47:41.953731 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:41.953698 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82e2cde4-77ff-4ae8-9161-5518af79e161-web-config" (OuterVolumeSpecName: "web-config") pod "82e2cde4-77ff-4ae8-9161-5518af79e161" (UID: "82e2cde4-77ff-4ae8-9161-5518af79e161"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 12 01:47:42.038297 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.038242 2579 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82e2cde4-77ff-4ae8-9161-5518af79e161-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 01:47:42.038297 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.038284 2579 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/82e2cde4-77ff-4ae8-9161-5518af79e161-config-out\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 01:47:42.038297 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.038293 2579 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/82e2cde4-77ff-4ae8-9161-5518af79e161-web-config\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 01:47:42.038297 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.038303 2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/82e2cde4-77ff-4ae8-9161-5518af79e161-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 01:47:42.038297 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.038314 2579 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/82e2cde4-77ff-4ae8-9161-5518af79e161-secret-kube-rbac-proxy\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 01:47:42.038645 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.038324 2579 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/82e2cde4-77ff-4ae8-9161-5518af79e161-secret-metrics-client-certs\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 01:47:42.038645 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.038332 2579 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/82e2cde4-77ff-4ae8-9161-5518af79e161-config\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 01:47:42.038645 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.038340 2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82e2cde4-77ff-4ae8-9161-5518af79e161-prometheus-trusted-ca-bundle\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 01:47:42.038645 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.038348 2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/82e2cde4-77ff-4ae8-9161-5518af79e161-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 01:47:42.038645 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.038357 2579 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/82e2cde4-77ff-4ae8-9161-5518af79e161-thanos-prometheus-http-client-file\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 01:47:42.038645 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.038365 2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/82e2cde4-77ff-4ae8-9161-5518af79e161-secret-prometheus-k8s-tls\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 01:47:42.038645 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.038374 2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/82e2cde4-77ff-4ae8-9161-5518af79e161-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 01:47:42.038645 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.038382 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgd5l\" (UniqueName: \"kubernetes.io/projected/82e2cde4-77ff-4ae8-9161-5518af79e161-kube-api-access-qgd5l\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 01:47:42.038645 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.038390 2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/82e2cde4-77ff-4ae8-9161-5518af79e161-prometheus-k8s-db\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 01:47:42.038645 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.038398 2579 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/82e2cde4-77ff-4ae8-9161-5518af79e161-tls-assets\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 01:47:42.038645 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.038406 2579 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/82e2cde4-77ff-4ae8-9161-5518af79e161-secret-grpc-tls\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 01:47:42.615362 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.615278 2579 generic.go:356] "Generic (PLEG): container finished" podID="82e2cde4-77ff-4ae8-9161-5518af79e161" containerID="a285bdb864f80206c67ed281cdb9d87bf400e11b36ce75b0cc9137921aeedaa5" exitCode=0 Sep 12 01:47:42.615362 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.615302 2579 generic.go:356] "Generic (PLEG): container finished" podID="82e2cde4-77ff-4ae8-9161-5518af79e161" containerID="f6d3434f6b051450df348d35f232a833ffa6047775f7f6e9d8926cd4f0b49de3" exitCode=0 Sep 12 01:47:42.615362 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.615310 2579 generic.go:356] "Generic (PLEG): container finished" podID="82e2cde4-77ff-4ae8-9161-5518af79e161" containerID="36989c40673921b740a77fd9502791e99a53bd2f5043c035845373d2b1a6b9f5" exitCode=0 Sep 12 01:47:42.615362 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.615320 2579 generic.go:356] "Generic (PLEG): container finished" podID="82e2cde4-77ff-4ae8-9161-5518af79e161" containerID="865594a92f3313da51183d5bbdb676c9f2881df9d541a8782d6101c25d488e73" exitCode=0 Sep 12 01:47:42.615362 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.615326 2579 generic.go:356] "Generic (PLEG): container finished" podID="82e2cde4-77ff-4ae8-9161-5518af79e161" containerID="8f952b20dab0ce83cb5d2e1c671ad7f0e791033d18d2a64f8e62adc43614939e" exitCode=0 Sep 12 01:47:42.615362 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.615333 2579 generic.go:356] "Generic (PLEG): container finished" podID="82e2cde4-77ff-4ae8-9161-5518af79e161" containerID="afcd635af5077b5fdd6e6174bba572ac1207d2c124e2b910d0fd86fc051f0760" exitCode=0 Sep 12 01:47:42.615362 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.615345 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"82e2cde4-77ff-4ae8-9161-5518af79e161","Type":"ContainerDied","Data":"a285bdb864f80206c67ed281cdb9d87bf400e11b36ce75b0cc9137921aeedaa5"} Sep 12 01:47:42.616017 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.615426 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"82e2cde4-77ff-4ae8-9161-5518af79e161","Type":"ContainerDied","Data":"f6d3434f6b051450df348d35f232a833ffa6047775f7f6e9d8926cd4f0b49de3"} Sep 12 01:47:42.616017 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.615436 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:42.616017 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.615478 2579 scope.go:117] "RemoveContainer" containerID="a285bdb864f80206c67ed281cdb9d87bf400e11b36ce75b0cc9137921aeedaa5" Sep 12 01:47:42.616017 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.615480 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"82e2cde4-77ff-4ae8-9161-5518af79e161","Type":"ContainerDied","Data":"36989c40673921b740a77fd9502791e99a53bd2f5043c035845373d2b1a6b9f5"} Sep 12 01:47:42.616017 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.615584 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"82e2cde4-77ff-4ae8-9161-5518af79e161","Type":"ContainerDied","Data":"865594a92f3313da51183d5bbdb676c9f2881df9d541a8782d6101c25d488e73"} Sep 12 01:47:42.616017 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.615618 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"82e2cde4-77ff-4ae8-9161-5518af79e161","Type":"ContainerDied","Data":"8f952b20dab0ce83cb5d2e1c671ad7f0e791033d18d2a64f8e62adc43614939e"} Sep 12 01:47:42.616017 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.615647 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"82e2cde4-77ff-4ae8-9161-5518af79e161","Type":"ContainerDied","Data":"afcd635af5077b5fdd6e6174bba572ac1207d2c124e2b910d0fd86fc051f0760"} Sep 12 01:47:42.616017 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.615671 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"82e2cde4-77ff-4ae8-9161-5518af79e161","Type":"ContainerDied","Data":"6fd2fe2a6db48658fc5c05d577a3a07531a261dbb9af1f102c27249f499a5148"} Sep 12 01:47:42.617059 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.617031 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-675d55d6d-6rkn4" event={"ID":"f746c621-3575-496b-88b5-eaf926350ab1","Type":"ContainerStarted","Data":"cd0e4dfb001592a2b9652a5821032c3b1d3d945ba63930efc593064e71e7b1bf"} Sep 12 01:47:42.624805 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.624784 2579 scope.go:117] "RemoveContainer" containerID="f6d3434f6b051450df348d35f232a833ffa6047775f7f6e9d8926cd4f0b49de3" Sep 12 01:47:42.638493 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.638471 2579 scope.go:117] "RemoveContainer" containerID="36989c40673921b740a77fd9502791e99a53bd2f5043c035845373d2b1a6b9f5" Sep 12 01:47:42.646314 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.646288 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 12 01:47:42.647128 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.647109 2579 scope.go:117] "RemoveContainer" containerID="865594a92f3313da51183d5bbdb676c9f2881df9d541a8782d6101c25d488e73" Sep 12 01:47:42.651929 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.651896 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 12 01:47:42.656265 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.656237 2579 scope.go:117] "RemoveContainer" containerID="8f952b20dab0ce83cb5d2e1c671ad7f0e791033d18d2a64f8e62adc43614939e" Sep 12 01:47:42.663902 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.663877 2579 scope.go:117] "RemoveContainer" containerID="afcd635af5077b5fdd6e6174bba572ac1207d2c124e2b910d0fd86fc051f0760" Sep 12 01:47:42.674642 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.674613 2579 scope.go:117] "RemoveContainer" containerID="428ac000e6d6d3ae1c1d8d02ac92ab43db09aada7b63dbd9f1243f791b71427e" Sep 12 01:47:42.674759 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.674726 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 12 01:47:42.675146 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.675129 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82e2cde4-77ff-4ae8-9161-5518af79e161" containerName="prometheus" Sep 12 01:47:42.675146 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.675146 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="82e2cde4-77ff-4ae8-9161-5518af79e161" containerName="prometheus" Sep 12 01:47:42.675284 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.675180 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82e2cde4-77ff-4ae8-9161-5518af79e161" containerName="kube-rbac-proxy-thanos" Sep 12 01:47:42.675284 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.675189 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="82e2cde4-77ff-4ae8-9161-5518af79e161" containerName="kube-rbac-proxy-thanos" Sep 12 01:47:42.675284 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.675204 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82e2cde4-77ff-4ae8-9161-5518af79e161" containerName="config-reloader" Sep 12 01:47:42.675284 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.675213 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="82e2cde4-77ff-4ae8-9161-5518af79e161" containerName="config-reloader" Sep 12 01:47:42.675284 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.675226 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82e2cde4-77ff-4ae8-9161-5518af79e161" containerName="kube-rbac-proxy" Sep 12 01:47:42.675284 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.675235 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="82e2cde4-77ff-4ae8-9161-5518af79e161" containerName="kube-rbac-proxy" Sep 12 01:47:42.675284 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.675253 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82e2cde4-77ff-4ae8-9161-5518af79e161" containerName="kube-rbac-proxy-web" Sep 12 01:47:42.675284 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.675261 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="82e2cde4-77ff-4ae8-9161-5518af79e161" containerName="kube-rbac-proxy-web" Sep 12 01:47:42.675284 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.675272 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82e2cde4-77ff-4ae8-9161-5518af79e161" containerName="thanos-sidecar" Sep 12 01:47:42.675284 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.675280 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="82e2cde4-77ff-4ae8-9161-5518af79e161" containerName="thanos-sidecar" Sep 12 01:47:42.675782 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.675292 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82e2cde4-77ff-4ae8-9161-5518af79e161" containerName="init-config-reloader" Sep 12 01:47:42.675782 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.675301 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="82e2cde4-77ff-4ae8-9161-5518af79e161" containerName="init-config-reloader" Sep 12 01:47:42.675782 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.675376 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="82e2cde4-77ff-4ae8-9161-5518af79e161" containerName="prometheus" Sep 12 01:47:42.675782 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.675395 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="82e2cde4-77ff-4ae8-9161-5518af79e161" containerName="kube-rbac-proxy" Sep 12 01:47:42.675782 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.675405 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="82e2cde4-77ff-4ae8-9161-5518af79e161" containerName="kube-rbac-proxy-web" Sep 12 01:47:42.675782 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.675416 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="82e2cde4-77ff-4ae8-9161-5518af79e161" containerName="kube-rbac-proxy-thanos" Sep 12 01:47:42.675782 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.675425 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="82e2cde4-77ff-4ae8-9161-5518af79e161" containerName="config-reloader" Sep 12 01:47:42.675782 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.675431 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="82e2cde4-77ff-4ae8-9161-5518af79e161" containerName="thanos-sidecar" Sep 12 01:47:42.680060 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.680038 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:42.682755 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.682734 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 12 01:47:42.682875 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.682771 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 12 01:47:42.682875 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.682787 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-q269p\"" Sep 12 01:47:42.683151 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.683113 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 12 01:47:42.683252 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.683152 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 12 01:47:42.683252 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.683197 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 12 01:47:42.683416 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.683395 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 12 01:47:42.683553 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.683426 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6bct4v63ttmfp\"" Sep 12 01:47:42.683553 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.683499 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 12 01:47:42.683915 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.683800 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 12 01:47:42.683915 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.683895 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 12 01:47:42.684077 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.683986 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 12 01:47:42.684270 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.684244 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 12 01:47:42.684507 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.684464 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 12 01:47:42.686370 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.686329 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 12 01:47:42.688641 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.688621 2579 scope.go:117] "RemoveContainer" containerID="a285bdb864f80206c67ed281cdb9d87bf400e11b36ce75b0cc9137921aeedaa5" Sep 12 01:47:42.689029 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:47:42.688998 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a285bdb864f80206c67ed281cdb9d87bf400e11b36ce75b0cc9137921aeedaa5\": container with ID starting with a285bdb864f80206c67ed281cdb9d87bf400e11b36ce75b0cc9137921aeedaa5 not found: ID does not exist" containerID="a285bdb864f80206c67ed281cdb9d87bf400e11b36ce75b0cc9137921aeedaa5" Sep 12 01:47:42.689094 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.689034 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a285bdb864f80206c67ed281cdb9d87bf400e11b36ce75b0cc9137921aeedaa5"} err="failed to get container status \"a285bdb864f80206c67ed281cdb9d87bf400e11b36ce75b0cc9137921aeedaa5\": rpc error: code = NotFound desc = could not find container \"a285bdb864f80206c67ed281cdb9d87bf400e11b36ce75b0cc9137921aeedaa5\": container with ID starting with a285bdb864f80206c67ed281cdb9d87bf400e11b36ce75b0cc9137921aeedaa5 not found: ID does not exist" Sep 12 01:47:42.689094 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.689058 2579 scope.go:117] "RemoveContainer" containerID="f6d3434f6b051450df348d35f232a833ffa6047775f7f6e9d8926cd4f0b49de3" Sep 12 01:47:42.689750 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.689588 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 12 01:47:42.689750 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:47:42.689656 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6d3434f6b051450df348d35f232a833ffa6047775f7f6e9d8926cd4f0b49de3\": container with ID starting with f6d3434f6b051450df348d35f232a833ffa6047775f7f6e9d8926cd4f0b49de3 not found: ID does not exist" containerID="f6d3434f6b051450df348d35f232a833ffa6047775f7f6e9d8926cd4f0b49de3" Sep 12 01:47:42.689750 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.689685 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6d3434f6b051450df348d35f232a833ffa6047775f7f6e9d8926cd4f0b49de3"} err="failed to get container status \"f6d3434f6b051450df348d35f232a833ffa6047775f7f6e9d8926cd4f0b49de3\": rpc error: code = NotFound desc = could not find container \"f6d3434f6b051450df348d35f232a833ffa6047775f7f6e9d8926cd4f0b49de3\": container with ID starting with f6d3434f6b051450df348d35f232a833ffa6047775f7f6e9d8926cd4f0b49de3 not found: ID does not exist" Sep 12 01:47:42.689750 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.689707 2579 scope.go:117] "RemoveContainer" containerID="36989c40673921b740a77fd9502791e99a53bd2f5043c035845373d2b1a6b9f5" Sep 12 01:47:42.690086 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:47:42.690062 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36989c40673921b740a77fd9502791e99a53bd2f5043c035845373d2b1a6b9f5\": container with ID starting with 36989c40673921b740a77fd9502791e99a53bd2f5043c035845373d2b1a6b9f5 not found: ID does not exist" containerID="36989c40673921b740a77fd9502791e99a53bd2f5043c035845373d2b1a6b9f5" Sep 12 01:47:42.690153 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.690095 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36989c40673921b740a77fd9502791e99a53bd2f5043c035845373d2b1a6b9f5"} err="failed to get container status \"36989c40673921b740a77fd9502791e99a53bd2f5043c035845373d2b1a6b9f5\": rpc error: code = NotFound desc = could not find container \"36989c40673921b740a77fd9502791e99a53bd2f5043c035845373d2b1a6b9f5\": container with ID starting with 36989c40673921b740a77fd9502791e99a53bd2f5043c035845373d2b1a6b9f5 not found: ID does not exist" Sep 12 01:47:42.690153 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.690111 2579 scope.go:117] "RemoveContainer" containerID="865594a92f3313da51183d5bbdb676c9f2881df9d541a8782d6101c25d488e73" Sep 12 01:47:42.690443 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:47:42.690419 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"865594a92f3313da51183d5bbdb676c9f2881df9d541a8782d6101c25d488e73\": container with ID starting with 865594a92f3313da51183d5bbdb676c9f2881df9d541a8782d6101c25d488e73 not found: ID does not exist" containerID="865594a92f3313da51183d5bbdb676c9f2881df9d541a8782d6101c25d488e73" Sep 12 01:47:42.690557 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.690505 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"865594a92f3313da51183d5bbdb676c9f2881df9d541a8782d6101c25d488e73"} err="failed to get container status \"865594a92f3313da51183d5bbdb676c9f2881df9d541a8782d6101c25d488e73\": rpc error: code = NotFound desc = could not find container \"865594a92f3313da51183d5bbdb676c9f2881df9d541a8782d6101c25d488e73\": container with ID starting with 865594a92f3313da51183d5bbdb676c9f2881df9d541a8782d6101c25d488e73 not found: ID does not exist" Sep 12 01:47:42.690557 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.690528 2579 scope.go:117] "RemoveContainer" containerID="8f952b20dab0ce83cb5d2e1c671ad7f0e791033d18d2a64f8e62adc43614939e" Sep 12 01:47:42.691007 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:47:42.690845 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f952b20dab0ce83cb5d2e1c671ad7f0e791033d18d2a64f8e62adc43614939e\": container with ID starting with 8f952b20dab0ce83cb5d2e1c671ad7f0e791033d18d2a64f8e62adc43614939e not found: ID does not exist" containerID="8f952b20dab0ce83cb5d2e1c671ad7f0e791033d18d2a64f8e62adc43614939e" Sep 12 01:47:42.691007 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.690877 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f952b20dab0ce83cb5d2e1c671ad7f0e791033d18d2a64f8e62adc43614939e"} err="failed to get container status \"8f952b20dab0ce83cb5d2e1c671ad7f0e791033d18d2a64f8e62adc43614939e\": rpc error: code = NotFound desc = could not find container \"8f952b20dab0ce83cb5d2e1c671ad7f0e791033d18d2a64f8e62adc43614939e\": container with ID starting with 8f952b20dab0ce83cb5d2e1c671ad7f0e791033d18d2a64f8e62adc43614939e not found: ID does not exist" Sep 12 01:47:42.691007 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.690899 2579 scope.go:117] "RemoveContainer" containerID="afcd635af5077b5fdd6e6174bba572ac1207d2c124e2b910d0fd86fc051f0760" Sep 12 01:47:42.692092 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:47:42.691686 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afcd635af5077b5fdd6e6174bba572ac1207d2c124e2b910d0fd86fc051f0760\": container with ID starting with afcd635af5077b5fdd6e6174bba572ac1207d2c124e2b910d0fd86fc051f0760 not found: ID does not exist" containerID="afcd635af5077b5fdd6e6174bba572ac1207d2c124e2b910d0fd86fc051f0760" Sep 12 01:47:42.692092 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.691733 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afcd635af5077b5fdd6e6174bba572ac1207d2c124e2b910d0fd86fc051f0760"} err="failed to get container status \"afcd635af5077b5fdd6e6174bba572ac1207d2c124e2b910d0fd86fc051f0760\": rpc error: code = NotFound desc = could not find container \"afcd635af5077b5fdd6e6174bba572ac1207d2c124e2b910d0fd86fc051f0760\": container with ID starting with afcd635af5077b5fdd6e6174bba572ac1207d2c124e2b910d0fd86fc051f0760 not found: ID does not exist" Sep 12 01:47:42.692092 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.691753 2579 scope.go:117] "RemoveContainer" containerID="428ac000e6d6d3ae1c1d8d02ac92ab43db09aada7b63dbd9f1243f791b71427e" Sep 12 01:47:42.692645 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:47:42.692620 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"428ac000e6d6d3ae1c1d8d02ac92ab43db09aada7b63dbd9f1243f791b71427e\": container with ID starting with 428ac000e6d6d3ae1c1d8d02ac92ab43db09aada7b63dbd9f1243f791b71427e not found: ID does not exist" containerID="428ac000e6d6d3ae1c1d8d02ac92ab43db09aada7b63dbd9f1243f791b71427e" Sep 12 01:47:42.692763 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.692650 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"428ac000e6d6d3ae1c1d8d02ac92ab43db09aada7b63dbd9f1243f791b71427e"} err="failed to get container status \"428ac000e6d6d3ae1c1d8d02ac92ab43db09aada7b63dbd9f1243f791b71427e\": rpc error: code = NotFound desc = could not find container \"428ac000e6d6d3ae1c1d8d02ac92ab43db09aada7b63dbd9f1243f791b71427e\": container with ID starting with 428ac000e6d6d3ae1c1d8d02ac92ab43db09aada7b63dbd9f1243f791b71427e not found: ID does not exist" Sep 12 01:47:42.692763 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.692669 2579 scope.go:117] "RemoveContainer" containerID="a285bdb864f80206c67ed281cdb9d87bf400e11b36ce75b0cc9137921aeedaa5" Sep 12 01:47:42.693056 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.693031 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a285bdb864f80206c67ed281cdb9d87bf400e11b36ce75b0cc9137921aeedaa5"} err="failed to get container status \"a285bdb864f80206c67ed281cdb9d87bf400e11b36ce75b0cc9137921aeedaa5\": rpc error: code = NotFound desc = could not find container \"a285bdb864f80206c67ed281cdb9d87bf400e11b36ce75b0cc9137921aeedaa5\": container with ID starting with a285bdb864f80206c67ed281cdb9d87bf400e11b36ce75b0cc9137921aeedaa5 not found: ID does not exist" Sep 12 01:47:42.693136 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.693060 2579 scope.go:117] "RemoveContainer" containerID="f6d3434f6b051450df348d35f232a833ffa6047775f7f6e9d8926cd4f0b49de3" Sep 12 01:47:42.694504 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.693964 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6d3434f6b051450df348d35f232a833ffa6047775f7f6e9d8926cd4f0b49de3"} err="failed to get container status \"f6d3434f6b051450df348d35f232a833ffa6047775f7f6e9d8926cd4f0b49de3\": rpc error: code = NotFound desc = could not find container \"f6d3434f6b051450df348d35f232a833ffa6047775f7f6e9d8926cd4f0b49de3\": container with ID starting with f6d3434f6b051450df348d35f232a833ffa6047775f7f6e9d8926cd4f0b49de3 not found: ID does not exist" Sep 12 01:47:42.694504 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.693994 2579 scope.go:117] "RemoveContainer" containerID="36989c40673921b740a77fd9502791e99a53bd2f5043c035845373d2b1a6b9f5" Sep 12 01:47:42.694504 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.694221 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 12 01:47:42.694504 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.694408 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36989c40673921b740a77fd9502791e99a53bd2f5043c035845373d2b1a6b9f5"} err="failed to get container status \"36989c40673921b740a77fd9502791e99a53bd2f5043c035845373d2b1a6b9f5\": rpc error: code = NotFound desc = could not find container \"36989c40673921b740a77fd9502791e99a53bd2f5043c035845373d2b1a6b9f5\": container with ID starting with 36989c40673921b740a77fd9502791e99a53bd2f5043c035845373d2b1a6b9f5 not found: ID does not exist" Sep 12 01:47:42.694504 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.694430 2579 scope.go:117] "RemoveContainer" containerID="865594a92f3313da51183d5bbdb676c9f2881df9d541a8782d6101c25d488e73" Sep 12 01:47:42.695085 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.695029 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"865594a92f3313da51183d5bbdb676c9f2881df9d541a8782d6101c25d488e73"} err="failed to get container status \"865594a92f3313da51183d5bbdb676c9f2881df9d541a8782d6101c25d488e73\": rpc error: code = NotFound desc = could not find container \"865594a92f3313da51183d5bbdb676c9f2881df9d541a8782d6101c25d488e73\": container with ID starting with 865594a92f3313da51183d5bbdb676c9f2881df9d541a8782d6101c25d488e73 not found: ID does not exist" Sep 12 01:47:42.695085 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.695075 2579 scope.go:117] "RemoveContainer" containerID="8f952b20dab0ce83cb5d2e1c671ad7f0e791033d18d2a64f8e62adc43614939e" Sep 12 01:47:42.695347 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.695321 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f952b20dab0ce83cb5d2e1c671ad7f0e791033d18d2a64f8e62adc43614939e"} err="failed to get container status \"8f952b20dab0ce83cb5d2e1c671ad7f0e791033d18d2a64f8e62adc43614939e\": rpc error: code = NotFound desc = could not find container \"8f952b20dab0ce83cb5d2e1c671ad7f0e791033d18d2a64f8e62adc43614939e\": container with ID starting with 8f952b20dab0ce83cb5d2e1c671ad7f0e791033d18d2a64f8e62adc43614939e not found: ID does not exist" Sep 12 01:47:42.695416 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.695350 2579 scope.go:117] "RemoveContainer" containerID="afcd635af5077b5fdd6e6174bba572ac1207d2c124e2b910d0fd86fc051f0760" Sep 12 01:47:42.695653 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.695625 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afcd635af5077b5fdd6e6174bba572ac1207d2c124e2b910d0fd86fc051f0760"} err="failed to get container status \"afcd635af5077b5fdd6e6174bba572ac1207d2c124e2b910d0fd86fc051f0760\": rpc error: code = NotFound desc = could not find container \"afcd635af5077b5fdd6e6174bba572ac1207d2c124e2b910d0fd86fc051f0760\": container with ID starting with afcd635af5077b5fdd6e6174bba572ac1207d2c124e2b910d0fd86fc051f0760 not found: ID does not exist" Sep 12 01:47:42.695797 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.695655 2579 scope.go:117] "RemoveContainer" containerID="428ac000e6d6d3ae1c1d8d02ac92ab43db09aada7b63dbd9f1243f791b71427e" Sep 12 01:47:42.695925 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.695899 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"428ac000e6d6d3ae1c1d8d02ac92ab43db09aada7b63dbd9f1243f791b71427e"} err="failed to get container status \"428ac000e6d6d3ae1c1d8d02ac92ab43db09aada7b63dbd9f1243f791b71427e\": rpc error: code = NotFound desc = could not find container \"428ac000e6d6d3ae1c1d8d02ac92ab43db09aada7b63dbd9f1243f791b71427e\": container with ID starting with 428ac000e6d6d3ae1c1d8d02ac92ab43db09aada7b63dbd9f1243f791b71427e not found: ID does not exist" Sep 12 01:47:42.695994 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.695926 2579 scope.go:117] "RemoveContainer" containerID="a285bdb864f80206c67ed281cdb9d87bf400e11b36ce75b0cc9137921aeedaa5" Sep 12 01:47:42.696869 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.696366 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a285bdb864f80206c67ed281cdb9d87bf400e11b36ce75b0cc9137921aeedaa5"} err="failed to get container status \"a285bdb864f80206c67ed281cdb9d87bf400e11b36ce75b0cc9137921aeedaa5\": rpc error: code = NotFound desc = could not find container \"a285bdb864f80206c67ed281cdb9d87bf400e11b36ce75b0cc9137921aeedaa5\": container with ID starting with a285bdb864f80206c67ed281cdb9d87bf400e11b36ce75b0cc9137921aeedaa5 not found: ID does not exist" Sep 12 01:47:42.696869 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.696387 2579 scope.go:117] "RemoveContainer" containerID="f6d3434f6b051450df348d35f232a833ffa6047775f7f6e9d8926cd4f0b49de3" Sep 12 01:47:42.696869 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.696738 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6d3434f6b051450df348d35f232a833ffa6047775f7f6e9d8926cd4f0b49de3"} err="failed to get container status \"f6d3434f6b051450df348d35f232a833ffa6047775f7f6e9d8926cd4f0b49de3\": rpc error: code = NotFound desc = could not find container \"f6d3434f6b051450df348d35f232a833ffa6047775f7f6e9d8926cd4f0b49de3\": container with ID starting with f6d3434f6b051450df348d35f232a833ffa6047775f7f6e9d8926cd4f0b49de3 not found: ID does not exist" Sep 12 01:47:42.696869 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.696760 2579 scope.go:117] "RemoveContainer" containerID="36989c40673921b740a77fd9502791e99a53bd2f5043c035845373d2b1a6b9f5" Sep 12 01:47:42.697111 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.697088 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36989c40673921b740a77fd9502791e99a53bd2f5043c035845373d2b1a6b9f5"} err="failed to get container status \"36989c40673921b740a77fd9502791e99a53bd2f5043c035845373d2b1a6b9f5\": rpc error: code = NotFound desc = could not find container \"36989c40673921b740a77fd9502791e99a53bd2f5043c035845373d2b1a6b9f5\": container with ID starting with 36989c40673921b740a77fd9502791e99a53bd2f5043c035845373d2b1a6b9f5 not found: ID does not exist" Sep 12 01:47:42.697155 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.697110 2579 scope.go:117] "RemoveContainer" containerID="865594a92f3313da51183d5bbdb676c9f2881df9d541a8782d6101c25d488e73" Sep 12 01:47:42.697506 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.697483 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"865594a92f3313da51183d5bbdb676c9f2881df9d541a8782d6101c25d488e73"} err="failed to get container status \"865594a92f3313da51183d5bbdb676c9f2881df9d541a8782d6101c25d488e73\": rpc error: code = NotFound desc = could not find container \"865594a92f3313da51183d5bbdb676c9f2881df9d541a8782d6101c25d488e73\": container with ID starting with 865594a92f3313da51183d5bbdb676c9f2881df9d541a8782d6101c25d488e73 not found: ID does not exist" Sep 12 01:47:42.697583 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.697507 2579 scope.go:117] "RemoveContainer" containerID="8f952b20dab0ce83cb5d2e1c671ad7f0e791033d18d2a64f8e62adc43614939e" Sep 12 01:47:42.697816 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.697779 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f952b20dab0ce83cb5d2e1c671ad7f0e791033d18d2a64f8e62adc43614939e"} err="failed to get container status \"8f952b20dab0ce83cb5d2e1c671ad7f0e791033d18d2a64f8e62adc43614939e\": rpc error: code = NotFound desc = could not find container \"8f952b20dab0ce83cb5d2e1c671ad7f0e791033d18d2a64f8e62adc43614939e\": container with ID starting with 8f952b20dab0ce83cb5d2e1c671ad7f0e791033d18d2a64f8e62adc43614939e not found: ID does not exist" Sep 12 01:47:42.697816 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.697803 2579 scope.go:117] "RemoveContainer" containerID="afcd635af5077b5fdd6e6174bba572ac1207d2c124e2b910d0fd86fc051f0760" Sep 12 01:47:42.698184 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.698143 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afcd635af5077b5fdd6e6174bba572ac1207d2c124e2b910d0fd86fc051f0760"} err="failed to get container status \"afcd635af5077b5fdd6e6174bba572ac1207d2c124e2b910d0fd86fc051f0760\": rpc error: code = NotFound desc = could not find container \"afcd635af5077b5fdd6e6174bba572ac1207d2c124e2b910d0fd86fc051f0760\": container with ID starting with afcd635af5077b5fdd6e6174bba572ac1207d2c124e2b910d0fd86fc051f0760 not found: ID does not exist" Sep 12 01:47:42.698184 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.698173 2579 scope.go:117] "RemoveContainer" containerID="428ac000e6d6d3ae1c1d8d02ac92ab43db09aada7b63dbd9f1243f791b71427e" Sep 12 01:47:42.698537 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.698515 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"428ac000e6d6d3ae1c1d8d02ac92ab43db09aada7b63dbd9f1243f791b71427e"} err="failed to get container status \"428ac000e6d6d3ae1c1d8d02ac92ab43db09aada7b63dbd9f1243f791b71427e\": rpc error: code = NotFound desc = could not find container \"428ac000e6d6d3ae1c1d8d02ac92ab43db09aada7b63dbd9f1243f791b71427e\": container with ID starting with 428ac000e6d6d3ae1c1d8d02ac92ab43db09aada7b63dbd9f1243f791b71427e not found: ID does not exist" Sep 12 01:47:42.698636 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.698538 2579 scope.go:117] "RemoveContainer" containerID="a285bdb864f80206c67ed281cdb9d87bf400e11b36ce75b0cc9137921aeedaa5" Sep 12 01:47:42.698819 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.698785 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a285bdb864f80206c67ed281cdb9d87bf400e11b36ce75b0cc9137921aeedaa5"} err="failed to get container status \"a285bdb864f80206c67ed281cdb9d87bf400e11b36ce75b0cc9137921aeedaa5\": rpc error: code = NotFound desc = could not find container \"a285bdb864f80206c67ed281cdb9d87bf400e11b36ce75b0cc9137921aeedaa5\": container with ID starting with a285bdb864f80206c67ed281cdb9d87bf400e11b36ce75b0cc9137921aeedaa5 not found: ID does not exist" Sep 12 01:47:42.698819 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.698810 2579 scope.go:117] "RemoveContainer" containerID="f6d3434f6b051450df348d35f232a833ffa6047775f7f6e9d8926cd4f0b49de3" Sep 12 01:47:42.699073 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.699048 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6d3434f6b051450df348d35f232a833ffa6047775f7f6e9d8926cd4f0b49de3"} err="failed to get container status \"f6d3434f6b051450df348d35f232a833ffa6047775f7f6e9d8926cd4f0b49de3\": rpc error: code = NotFound desc = could not find container \"f6d3434f6b051450df348d35f232a833ffa6047775f7f6e9d8926cd4f0b49de3\": container with ID starting with f6d3434f6b051450df348d35f232a833ffa6047775f7f6e9d8926cd4f0b49de3 not found: ID does not exist" Sep 12 01:47:42.699073 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.699073 2579 scope.go:117] "RemoveContainer" containerID="36989c40673921b740a77fd9502791e99a53bd2f5043c035845373d2b1a6b9f5" Sep 12 01:47:42.699404 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.699382 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36989c40673921b740a77fd9502791e99a53bd2f5043c035845373d2b1a6b9f5"} err="failed to get container status \"36989c40673921b740a77fd9502791e99a53bd2f5043c035845373d2b1a6b9f5\": rpc error: code = NotFound desc = could not find container \"36989c40673921b740a77fd9502791e99a53bd2f5043c035845373d2b1a6b9f5\": container with ID starting with 36989c40673921b740a77fd9502791e99a53bd2f5043c035845373d2b1a6b9f5 not found: ID does not exist" Sep 12 01:47:42.699521 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.699405 2579 scope.go:117] "RemoveContainer" containerID="865594a92f3313da51183d5bbdb676c9f2881df9d541a8782d6101c25d488e73" Sep 12 01:47:42.699764 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.699743 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"865594a92f3313da51183d5bbdb676c9f2881df9d541a8782d6101c25d488e73"} err="failed to get container status \"865594a92f3313da51183d5bbdb676c9f2881df9d541a8782d6101c25d488e73\": rpc error: code = NotFound desc = could not find container \"865594a92f3313da51183d5bbdb676c9f2881df9d541a8782d6101c25d488e73\": container with ID starting with 865594a92f3313da51183d5bbdb676c9f2881df9d541a8782d6101c25d488e73 not found: ID does not exist" Sep 12 01:47:42.699843 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.699765 2579 scope.go:117] "RemoveContainer" containerID="8f952b20dab0ce83cb5d2e1c671ad7f0e791033d18d2a64f8e62adc43614939e" Sep 12 01:47:42.700030 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.700008 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f952b20dab0ce83cb5d2e1c671ad7f0e791033d18d2a64f8e62adc43614939e"} err="failed to get container status \"8f952b20dab0ce83cb5d2e1c671ad7f0e791033d18d2a64f8e62adc43614939e\": rpc error: code = NotFound desc = could not find container \"8f952b20dab0ce83cb5d2e1c671ad7f0e791033d18d2a64f8e62adc43614939e\": container with ID starting with 8f952b20dab0ce83cb5d2e1c671ad7f0e791033d18d2a64f8e62adc43614939e not found: ID does not exist" Sep 12 01:47:42.700089 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.700033 2579 scope.go:117] "RemoveContainer" containerID="afcd635af5077b5fdd6e6174bba572ac1207d2c124e2b910d0fd86fc051f0760" Sep 12 01:47:42.700563 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.700464 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afcd635af5077b5fdd6e6174bba572ac1207d2c124e2b910d0fd86fc051f0760"} err="failed to get container status \"afcd635af5077b5fdd6e6174bba572ac1207d2c124e2b910d0fd86fc051f0760\": rpc error: code = NotFound desc = could not find container \"afcd635af5077b5fdd6e6174bba572ac1207d2c124e2b910d0fd86fc051f0760\": container with ID starting with afcd635af5077b5fdd6e6174bba572ac1207d2c124e2b910d0fd86fc051f0760 not found: ID does not exist" Sep 12 01:47:42.700563 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.700491 2579 scope.go:117] "RemoveContainer" containerID="428ac000e6d6d3ae1c1d8d02ac92ab43db09aada7b63dbd9f1243f791b71427e" Sep 12 01:47:42.700774 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.700749 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"428ac000e6d6d3ae1c1d8d02ac92ab43db09aada7b63dbd9f1243f791b71427e"} err="failed to get container status \"428ac000e6d6d3ae1c1d8d02ac92ab43db09aada7b63dbd9f1243f791b71427e\": rpc error: code = NotFound desc = could not find container \"428ac000e6d6d3ae1c1d8d02ac92ab43db09aada7b63dbd9f1243f791b71427e\": container with ID starting with 428ac000e6d6d3ae1c1d8d02ac92ab43db09aada7b63dbd9f1243f791b71427e not found: ID does not exist" Sep 12 01:47:42.700774 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.700774 2579 scope.go:117] "RemoveContainer" containerID="a285bdb864f80206c67ed281cdb9d87bf400e11b36ce75b0cc9137921aeedaa5" Sep 12 01:47:42.701037 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.701018 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a285bdb864f80206c67ed281cdb9d87bf400e11b36ce75b0cc9137921aeedaa5"} err="failed to get container status \"a285bdb864f80206c67ed281cdb9d87bf400e11b36ce75b0cc9137921aeedaa5\": rpc error: code = NotFound desc = could not find container \"a285bdb864f80206c67ed281cdb9d87bf400e11b36ce75b0cc9137921aeedaa5\": container with ID starting with a285bdb864f80206c67ed281cdb9d87bf400e11b36ce75b0cc9137921aeedaa5 not found: ID does not exist" Sep 12 01:47:42.701109 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.701038 2579 scope.go:117] "RemoveContainer" containerID="f6d3434f6b051450df348d35f232a833ffa6047775f7f6e9d8926cd4f0b49de3" Sep 12 01:47:42.701309 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.701286 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6d3434f6b051450df348d35f232a833ffa6047775f7f6e9d8926cd4f0b49de3"} err="failed to get container status \"f6d3434f6b051450df348d35f232a833ffa6047775f7f6e9d8926cd4f0b49de3\": rpc error: code = NotFound desc = could not find container \"f6d3434f6b051450df348d35f232a833ffa6047775f7f6e9d8926cd4f0b49de3\": container with ID starting with f6d3434f6b051450df348d35f232a833ffa6047775f7f6e9d8926cd4f0b49de3 not found: ID does not exist" Sep 12 01:47:42.701382 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.701311 2579 scope.go:117] "RemoveContainer" containerID="36989c40673921b740a77fd9502791e99a53bd2f5043c035845373d2b1a6b9f5" Sep 12 01:47:42.701608 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.701588 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36989c40673921b740a77fd9502791e99a53bd2f5043c035845373d2b1a6b9f5"} err="failed to get container status \"36989c40673921b740a77fd9502791e99a53bd2f5043c035845373d2b1a6b9f5\": rpc error: code = NotFound desc = could not find container \"36989c40673921b740a77fd9502791e99a53bd2f5043c035845373d2b1a6b9f5\": container with ID starting with 36989c40673921b740a77fd9502791e99a53bd2f5043c035845373d2b1a6b9f5 not found: ID does not exist" Sep 12 01:47:42.701682 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.701610 2579 scope.go:117] "RemoveContainer" containerID="865594a92f3313da51183d5bbdb676c9f2881df9d541a8782d6101c25d488e73" Sep 12 01:47:42.701894 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.701872 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"865594a92f3313da51183d5bbdb676c9f2881df9d541a8782d6101c25d488e73"} err="failed to get container status \"865594a92f3313da51183d5bbdb676c9f2881df9d541a8782d6101c25d488e73\": rpc error: code = NotFound desc = could not find container \"865594a92f3313da51183d5bbdb676c9f2881df9d541a8782d6101c25d488e73\": container with ID starting with 865594a92f3313da51183d5bbdb676c9f2881df9d541a8782d6101c25d488e73 not found: ID does not exist" Sep 12 01:47:42.702013 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.701896 2579 scope.go:117] "RemoveContainer" containerID="8f952b20dab0ce83cb5d2e1c671ad7f0e791033d18d2a64f8e62adc43614939e" Sep 12 01:47:42.702180 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.702161 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f952b20dab0ce83cb5d2e1c671ad7f0e791033d18d2a64f8e62adc43614939e"} err="failed to get container status \"8f952b20dab0ce83cb5d2e1c671ad7f0e791033d18d2a64f8e62adc43614939e\": rpc error: code = NotFound desc = could not find container \"8f952b20dab0ce83cb5d2e1c671ad7f0e791033d18d2a64f8e62adc43614939e\": container with ID starting with 8f952b20dab0ce83cb5d2e1c671ad7f0e791033d18d2a64f8e62adc43614939e not found: ID does not exist" Sep 12 01:47:42.702180 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.702181 2579 scope.go:117] "RemoveContainer" containerID="afcd635af5077b5fdd6e6174bba572ac1207d2c124e2b910d0fd86fc051f0760" Sep 12 01:47:42.702713 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.702418 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afcd635af5077b5fdd6e6174bba572ac1207d2c124e2b910d0fd86fc051f0760"} err="failed to get container status \"afcd635af5077b5fdd6e6174bba572ac1207d2c124e2b910d0fd86fc051f0760\": rpc error: code = NotFound desc = could not find container \"afcd635af5077b5fdd6e6174bba572ac1207d2c124e2b910d0fd86fc051f0760\": container with ID starting with afcd635af5077b5fdd6e6174bba572ac1207d2c124e2b910d0fd86fc051f0760 not found: ID does not exist" Sep 12 01:47:42.702713 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.702438 2579 scope.go:117] "RemoveContainer" containerID="428ac000e6d6d3ae1c1d8d02ac92ab43db09aada7b63dbd9f1243f791b71427e" Sep 12 01:47:42.702841 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.702731 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"428ac000e6d6d3ae1c1d8d02ac92ab43db09aada7b63dbd9f1243f791b71427e"} err="failed to get container status \"428ac000e6d6d3ae1c1d8d02ac92ab43db09aada7b63dbd9f1243f791b71427e\": rpc error: code = NotFound desc = could not find container \"428ac000e6d6d3ae1c1d8d02ac92ab43db09aada7b63dbd9f1243f791b71427e\": container with ID starting with 428ac000e6d6d3ae1c1d8d02ac92ab43db09aada7b63dbd9f1243f791b71427e not found: ID does not exist" Sep 12 01:47:42.702841 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.702750 2579 scope.go:117] "RemoveContainer" containerID="a285bdb864f80206c67ed281cdb9d87bf400e11b36ce75b0cc9137921aeedaa5" Sep 12 01:47:42.703134 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.703041 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a285bdb864f80206c67ed281cdb9d87bf400e11b36ce75b0cc9137921aeedaa5"} err="failed to get container status \"a285bdb864f80206c67ed281cdb9d87bf400e11b36ce75b0cc9137921aeedaa5\": rpc error: code = NotFound desc = could not find container \"a285bdb864f80206c67ed281cdb9d87bf400e11b36ce75b0cc9137921aeedaa5\": container with ID starting with a285bdb864f80206c67ed281cdb9d87bf400e11b36ce75b0cc9137921aeedaa5 not found: ID does not exist" Sep 12 01:47:42.703134 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.703065 2579 scope.go:117] "RemoveContainer" containerID="f6d3434f6b051450df348d35f232a833ffa6047775f7f6e9d8926cd4f0b49de3" Sep 12 01:47:42.703425 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.703334 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6d3434f6b051450df348d35f232a833ffa6047775f7f6e9d8926cd4f0b49de3"} err="failed to get container status \"f6d3434f6b051450df348d35f232a833ffa6047775f7f6e9d8926cd4f0b49de3\": rpc error: code = NotFound desc = could not find container \"f6d3434f6b051450df348d35f232a833ffa6047775f7f6e9d8926cd4f0b49de3\": container with ID starting with f6d3434f6b051450df348d35f232a833ffa6047775f7f6e9d8926cd4f0b49de3 not found: ID does not exist" Sep 12 01:47:42.703425 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.703360 2579 scope.go:117] "RemoveContainer" containerID="36989c40673921b740a77fd9502791e99a53bd2f5043c035845373d2b1a6b9f5" Sep 12 01:47:42.703667 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.703636 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36989c40673921b740a77fd9502791e99a53bd2f5043c035845373d2b1a6b9f5"} err="failed to get container status \"36989c40673921b740a77fd9502791e99a53bd2f5043c035845373d2b1a6b9f5\": rpc error: code = NotFound desc = could not find container \"36989c40673921b740a77fd9502791e99a53bd2f5043c035845373d2b1a6b9f5\": container with ID starting with 36989c40673921b740a77fd9502791e99a53bd2f5043c035845373d2b1a6b9f5 not found: ID does not exist" Sep 12 01:47:42.703736 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.703669 2579 scope.go:117] "RemoveContainer" containerID="865594a92f3313da51183d5bbdb676c9f2881df9d541a8782d6101c25d488e73" Sep 12 01:47:42.704046 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.703951 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"865594a92f3313da51183d5bbdb676c9f2881df9d541a8782d6101c25d488e73"} err="failed to get container status \"865594a92f3313da51183d5bbdb676c9f2881df9d541a8782d6101c25d488e73\": rpc error: code = NotFound desc = could not find container \"865594a92f3313da51183d5bbdb676c9f2881df9d541a8782d6101c25d488e73\": container with ID starting with 865594a92f3313da51183d5bbdb676c9f2881df9d541a8782d6101c25d488e73 not found: ID does not exist" Sep 12 01:47:42.704046 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.703977 2579 scope.go:117] "RemoveContainer" containerID="8f952b20dab0ce83cb5d2e1c671ad7f0e791033d18d2a64f8e62adc43614939e" Sep 12 01:47:42.704387 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.704352 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f952b20dab0ce83cb5d2e1c671ad7f0e791033d18d2a64f8e62adc43614939e"} err="failed to get container status \"8f952b20dab0ce83cb5d2e1c671ad7f0e791033d18d2a64f8e62adc43614939e\": rpc error: code = NotFound desc = could not find container \"8f952b20dab0ce83cb5d2e1c671ad7f0e791033d18d2a64f8e62adc43614939e\": container with ID starting with 8f952b20dab0ce83cb5d2e1c671ad7f0e791033d18d2a64f8e62adc43614939e not found: ID does not exist" Sep 12 01:47:42.704387 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.704388 2579 scope.go:117] "RemoveContainer" containerID="afcd635af5077b5fdd6e6174bba572ac1207d2c124e2b910d0fd86fc051f0760" Sep 12 01:47:42.704796 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.704771 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afcd635af5077b5fdd6e6174bba572ac1207d2c124e2b910d0fd86fc051f0760"} err="failed to get container status \"afcd635af5077b5fdd6e6174bba572ac1207d2c124e2b910d0fd86fc051f0760\": rpc error: code = NotFound desc = could not find container \"afcd635af5077b5fdd6e6174bba572ac1207d2c124e2b910d0fd86fc051f0760\": container with ID starting with afcd635af5077b5fdd6e6174bba572ac1207d2c124e2b910d0fd86fc051f0760 not found: ID does not exist" Sep 12 01:47:42.704796 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.704796 2579 scope.go:117] "RemoveContainer" containerID="428ac000e6d6d3ae1c1d8d02ac92ab43db09aada7b63dbd9f1243f791b71427e" Sep 12 01:47:42.705153 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.705132 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"428ac000e6d6d3ae1c1d8d02ac92ab43db09aada7b63dbd9f1243f791b71427e"} err="failed to get container status \"428ac000e6d6d3ae1c1d8d02ac92ab43db09aada7b63dbd9f1243f791b71427e\": rpc error: code = NotFound desc = could not find container \"428ac000e6d6d3ae1c1d8d02ac92ab43db09aada7b63dbd9f1243f791b71427e\": container with ID starting with 428ac000e6d6d3ae1c1d8d02ac92ab43db09aada7b63dbd9f1243f791b71427e not found: ID does not exist" Sep 12 01:47:42.744078 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.744039 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f40b713-de7e-443d-bd28-0bf631ee73b3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7f40b713-de7e-443d-bd28-0bf631ee73b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:42.744273 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.744135 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7f40b713-de7e-443d-bd28-0bf631ee73b3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7f40b713-de7e-443d-bd28-0bf631ee73b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:42.744273 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.744169 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7f40b713-de7e-443d-bd28-0bf631ee73b3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7f40b713-de7e-443d-bd28-0bf631ee73b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:42.744273 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.744220 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7f40b713-de7e-443d-bd28-0bf631ee73b3-config-out\") pod \"prometheus-k8s-0\" (UID: \"7f40b713-de7e-443d-bd28-0bf631ee73b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:42.744273 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.744242 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7f40b713-de7e-443d-bd28-0bf631ee73b3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7f40b713-de7e-443d-bd28-0bf631ee73b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:42.744560 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.744280 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7f40b713-de7e-443d-bd28-0bf631ee73b3-config\") pod \"prometheus-k8s-0\" (UID: \"7f40b713-de7e-443d-bd28-0bf631ee73b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:42.744560 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.744309 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f40b713-de7e-443d-bd28-0bf631ee73b3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7f40b713-de7e-443d-bd28-0bf631ee73b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:42.744560 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.744338 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7f40b713-de7e-443d-bd28-0bf631ee73b3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7f40b713-de7e-443d-bd28-0bf631ee73b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:42.744560 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.744407 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7f40b713-de7e-443d-bd28-0bf631ee73b3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7f40b713-de7e-443d-bd28-0bf631ee73b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:42.744560 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.744500 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7f40b713-de7e-443d-bd28-0bf631ee73b3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7f40b713-de7e-443d-bd28-0bf631ee73b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:42.744951 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.744935 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7f40b713-de7e-443d-bd28-0bf631ee73b3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7f40b713-de7e-443d-bd28-0bf631ee73b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:42.745270 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.745244 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7f40b713-de7e-443d-bd28-0bf631ee73b3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7f40b713-de7e-443d-bd28-0bf631ee73b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:42.745339 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.745320 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7f40b713-de7e-443d-bd28-0bf631ee73b3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7f40b713-de7e-443d-bd28-0bf631ee73b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:42.745395 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.745357 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7f40b713-de7e-443d-bd28-0bf631ee73b3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7f40b713-de7e-443d-bd28-0bf631ee73b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:42.745473 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.745406 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7f40b713-de7e-443d-bd28-0bf631ee73b3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7f40b713-de7e-443d-bd28-0bf631ee73b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:42.745473 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.745433 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f40b713-de7e-443d-bd28-0bf631ee73b3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7f40b713-de7e-443d-bd28-0bf631ee73b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:42.745579 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.745537 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6cwg\" (UniqueName: \"kubernetes.io/projected/7f40b713-de7e-443d-bd28-0bf631ee73b3-kube-api-access-w6cwg\") pod \"prometheus-k8s-0\" (UID: \"7f40b713-de7e-443d-bd28-0bf631ee73b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:42.745638 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.745582 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7f40b713-de7e-443d-bd28-0bf631ee73b3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7f40b713-de7e-443d-bd28-0bf631ee73b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:42.745638 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.745612 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7f40b713-de7e-443d-bd28-0bf631ee73b3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7f40b713-de7e-443d-bd28-0bf631ee73b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:42.745736 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.745635 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7f40b713-de7e-443d-bd28-0bf631ee73b3-web-config\") pod \"prometheus-k8s-0\" (UID: \"7f40b713-de7e-443d-bd28-0bf631ee73b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:42.846212 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.846176 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7f40b713-de7e-443d-bd28-0bf631ee73b3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7f40b713-de7e-443d-bd28-0bf631ee73b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:42.846212 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.846218 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7f40b713-de7e-443d-bd28-0bf631ee73b3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7f40b713-de7e-443d-bd28-0bf631ee73b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:42.846471 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.846251 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7f40b713-de7e-443d-bd28-0bf631ee73b3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7f40b713-de7e-443d-bd28-0bf631ee73b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:42.846471 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.846279 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7f40b713-de7e-443d-bd28-0bf631ee73b3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7f40b713-de7e-443d-bd28-0bf631ee73b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:42.846471 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.846301 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7f40b713-de7e-443d-bd28-0bf631ee73b3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7f40b713-de7e-443d-bd28-0bf631ee73b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:42.846471 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.846317 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7f40b713-de7e-443d-bd28-0bf631ee73b3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7f40b713-de7e-443d-bd28-0bf631ee73b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:42.846471 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.846355 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7f40b713-de7e-443d-bd28-0bf631ee73b3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7f40b713-de7e-443d-bd28-0bf631ee73b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:42.846725 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.846670 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7f40b713-de7e-443d-bd28-0bf631ee73b3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7f40b713-de7e-443d-bd28-0bf631ee73b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:42.847069 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.847047 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7f40b713-de7e-443d-bd28-0bf631ee73b3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7f40b713-de7e-443d-bd28-0bf631ee73b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:42.847220 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.847179 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7f40b713-de7e-443d-bd28-0bf631ee73b3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7f40b713-de7e-443d-bd28-0bf631ee73b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:42.847368 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.847249 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f40b713-de7e-443d-bd28-0bf631ee73b3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7f40b713-de7e-443d-bd28-0bf631ee73b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:42.847368 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.847312 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6cwg\" (UniqueName: \"kubernetes.io/projected/7f40b713-de7e-443d-bd28-0bf631ee73b3-kube-api-access-w6cwg\") pod \"prometheus-k8s-0\" (UID: \"7f40b713-de7e-443d-bd28-0bf631ee73b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:42.847368 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.847343 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7f40b713-de7e-443d-bd28-0bf631ee73b3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7f40b713-de7e-443d-bd28-0bf631ee73b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:42.847368 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.847373 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7f40b713-de7e-443d-bd28-0bf631ee73b3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7f40b713-de7e-443d-bd28-0bf631ee73b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:42.847368 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.847396 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7f40b713-de7e-443d-bd28-0bf631ee73b3-web-config\") pod \"prometheus-k8s-0\" (UID: \"7f40b713-de7e-443d-bd28-0bf631ee73b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:42.847687 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.847428 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f40b713-de7e-443d-bd28-0bf631ee73b3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7f40b713-de7e-443d-bd28-0bf631ee73b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:42.847687 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.847490 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7f40b713-de7e-443d-bd28-0bf631ee73b3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7f40b713-de7e-443d-bd28-0bf631ee73b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:42.847687 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.847519 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7f40b713-de7e-443d-bd28-0bf631ee73b3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7f40b713-de7e-443d-bd28-0bf631ee73b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:42.847687 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.847559 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7f40b713-de7e-443d-bd28-0bf631ee73b3-config-out\") pod \"prometheus-k8s-0\" (UID: \"7f40b713-de7e-443d-bd28-0bf631ee73b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:42.847687 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.847580 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7f40b713-de7e-443d-bd28-0bf631ee73b3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7f40b713-de7e-443d-bd28-0bf631ee73b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:42.847687 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.847604 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7f40b713-de7e-443d-bd28-0bf631ee73b3-config\") pod \"prometheus-k8s-0\" (UID: \"7f40b713-de7e-443d-bd28-0bf631ee73b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:42.847687 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.847631 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f40b713-de7e-443d-bd28-0bf631ee73b3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7f40b713-de7e-443d-bd28-0bf631ee73b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:42.847687 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.847656 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7f40b713-de7e-443d-bd28-0bf631ee73b3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7f40b713-de7e-443d-bd28-0bf631ee73b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:42.848336 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.848312 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f40b713-de7e-443d-bd28-0bf631ee73b3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7f40b713-de7e-443d-bd28-0bf631ee73b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:42.850683 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.850035 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7f40b713-de7e-443d-bd28-0bf631ee73b3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7f40b713-de7e-443d-bd28-0bf631ee73b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:42.850683 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.850155 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7f40b713-de7e-443d-bd28-0bf631ee73b3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7f40b713-de7e-443d-bd28-0bf631ee73b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:42.850683 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.850505 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7f40b713-de7e-443d-bd28-0bf631ee73b3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7f40b713-de7e-443d-bd28-0bf631ee73b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:42.850899 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.850686 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7f40b713-de7e-443d-bd28-0bf631ee73b3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7f40b713-de7e-443d-bd28-0bf631ee73b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:42.851316 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.851112 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7f40b713-de7e-443d-bd28-0bf631ee73b3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7f40b713-de7e-443d-bd28-0bf631ee73b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:42.853712 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.852773 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7f40b713-de7e-443d-bd28-0bf631ee73b3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7f40b713-de7e-443d-bd28-0bf631ee73b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:42.853712 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.853369 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7f40b713-de7e-443d-bd28-0bf631ee73b3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7f40b713-de7e-443d-bd28-0bf631ee73b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:42.853712 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.853660 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/7f40b713-de7e-443d-bd28-0bf631ee73b3-config\") pod \"prometheus-k8s-0\" (UID: \"7f40b713-de7e-443d-bd28-0bf631ee73b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:42.853712 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.853669 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f40b713-de7e-443d-bd28-0bf631ee73b3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7f40b713-de7e-443d-bd28-0bf631ee73b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:42.853978 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.853718 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7f40b713-de7e-443d-bd28-0bf631ee73b3-config-out\") pod \"prometheus-k8s-0\" (UID: \"7f40b713-de7e-443d-bd28-0bf631ee73b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:42.853978 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.853953 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7f40b713-de7e-443d-bd28-0bf631ee73b3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7f40b713-de7e-443d-bd28-0bf631ee73b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:42.854085 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.854051 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7f40b713-de7e-443d-bd28-0bf631ee73b3-web-config\") pod \"prometheus-k8s-0\" (UID: \"7f40b713-de7e-443d-bd28-0bf631ee73b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:42.854242 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.854215 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7f40b713-de7e-443d-bd28-0bf631ee73b3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7f40b713-de7e-443d-bd28-0bf631ee73b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:42.854751 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.854710 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f40b713-de7e-443d-bd28-0bf631ee73b3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7f40b713-de7e-443d-bd28-0bf631ee73b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:42.855643 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.855622 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7f40b713-de7e-443d-bd28-0bf631ee73b3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7f40b713-de7e-443d-bd28-0bf631ee73b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:42.856678 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.856659 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6cwg\" (UniqueName: \"kubernetes.io/projected/7f40b713-de7e-443d-bd28-0bf631ee73b3-kube-api-access-w6cwg\") pod \"prometheus-k8s-0\" (UID: \"7f40b713-de7e-443d-bd28-0bf631ee73b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:42.997522 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:42.997486 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:43.299019 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:43.298994 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 12 01:47:43.623373 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:47:43.623339 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f40b713_de7e_443d_bd28_0bf631ee73b3.slice/crio-eaf880b34f12afdfc38c8b550f277255f347083223d625cc208efb8b32368a40 WatchSource:0}: Error finding container eaf880b34f12afdfc38c8b550f277255f347083223d625cc208efb8b32368a40: Status 404 returned error can't find the container with id eaf880b34f12afdfc38c8b550f277255f347083223d625cc208efb8b32368a40 Sep 12 01:47:43.727574 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:43.727547 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82e2cde4-77ff-4ae8-9161-5518af79e161" path="/var/lib/kubelet/pods/82e2cde4-77ff-4ae8-9161-5518af79e161/volumes" Sep 12 01:47:44.637955 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:44.637921 2579 generic.go:356] "Generic (PLEG): container finished" podID="7f40b713-de7e-443d-bd28-0bf631ee73b3" containerID="5eb9d27115914a5c97fed719248b619cfc702bc8b081deb18f3024a7f16a79de" exitCode=0 Sep 12 01:47:44.638385 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:44.638004 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7f40b713-de7e-443d-bd28-0bf631ee73b3","Type":"ContainerDied","Data":"5eb9d27115914a5c97fed719248b619cfc702bc8b081deb18f3024a7f16a79de"} Sep 12 01:47:44.638385 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:44.638092 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7f40b713-de7e-443d-bd28-0bf631ee73b3","Type":"ContainerStarted","Data":"eaf880b34f12afdfc38c8b550f277255f347083223d625cc208efb8b32368a40"} Sep 12 01:47:44.640186 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:44.640162 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-675d55d6d-6rkn4" event={"ID":"f746c621-3575-496b-88b5-eaf926350ab1","Type":"ContainerStarted","Data":"bb96f041c076c789a936d6d93e9a50439b87be6da8ff1c4ce28aa012e3f3092d"} Sep 12 01:47:44.640302 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:44.640207 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-675d55d6d-6rkn4" event={"ID":"f746c621-3575-496b-88b5-eaf926350ab1","Type":"ContainerStarted","Data":"28afd9a8e072982162e7ad07212d0aed072dcb8b094345a21bb5bb9046d8110f"} Sep 12 01:47:44.640302 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:44.640226 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-675d55d6d-6rkn4" event={"ID":"f746c621-3575-496b-88b5-eaf926350ab1","Type":"ContainerStarted","Data":"143ce59aecdfb00733984f5daf6f371f8857e9cfa32e57ae3f364c1b3bfb8d98"} Sep 12 01:47:44.685171 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:44.685119 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-675d55d6d-6rkn4" podStartSLOduration=1.907788604 podStartE2EDuration="3.685102182s" totalImagesPullingTime="1.777313578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:47:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:47:44.683639564 +0000 UTC m=+167.475633359" watchObservedRunningTime="2026-09-12 01:47:44.685102182 +0000 UTC m=+167.477095977" Sep 12 01:47:44.832428 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:44.832402 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:47:44.932016 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:44.931924 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:47:45.650156 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:45.650118 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7f40b713-de7e-443d-bd28-0bf631ee73b3","Type":"ContainerStarted","Data":"1ec784642c6bbf52711f0e5bfa761ac603623b7a5a0a835202ce6121f32a3b4f"} Sep 12 01:47:45.650607 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:45.650207 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7f40b713-de7e-443d-bd28-0bf631ee73b3","Type":"ContainerStarted","Data":"582ca34b762cda2349a098db67d874ed9acecbdd207f2d161ea32d29d4561cbf"} Sep 12 01:47:45.650607 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:45.650255 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7f40b713-de7e-443d-bd28-0bf631ee73b3","Type":"ContainerStarted","Data":"e70848a9caed05891d50166a48114cba260c728e20bd4cdb0efd48f6fcf249e3"} Sep 12 01:47:45.650607 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:45.650297 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7f40b713-de7e-443d-bd28-0bf631ee73b3","Type":"ContainerStarted","Data":"f3bba5fc8c0680259dda3588dd8f7405f3c2088e182d9e5f6b84c331b5062918"} Sep 12 01:47:45.650607 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:45.650336 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7f40b713-de7e-443d-bd28-0bf631ee73b3","Type":"ContainerStarted","Data":"55db8b9f345883e0950a110a62eed53bea9420e212bc5e458f353539baa4f89a"} Sep 12 01:47:45.650607 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:45.650378 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7f40b713-de7e-443d-bd28-0bf631ee73b3","Type":"ContainerStarted","Data":"dfe0c2bcdc7ff90d0b5928253d74d119abe6758d293cd13e196407d1365a0e09"} Sep 12 01:47:45.678847 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:45.678791 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.656586533 podStartE2EDuration="3.678763827s" totalImagesPullingTime="0s" totalInitContainerRuntime="22.177294ms" isStatefulPod=false podCreationTimestamp="2026-09-12 01:47:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:47:45.67703796 +0000 UTC m=+168.469031777" watchObservedRunningTime="2026-09-12 01:47:45.678763827 +0000 UTC m=+168.470757623" Sep 12 01:47:47.997835 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:47.997780 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:47:56.103270 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:56.103175 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:47:56.155896 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:56.155853 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:47:56.213325 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:47:56.213288 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:48:10.224467 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:48:10.224422 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:48:42.998666 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:48:42.998623 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:48:43.017754 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:48:43.017729 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:48:43.909445 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:48:43.909419 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 12 01:48:51.402247 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:48:51.402207 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:48:51.503659 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:48:51.503627 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:48:52.574242 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:48:52.574192 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:48:52.638436 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:48:52.638400 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:48:52.721011 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:48:52.720979 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:48:52.821580 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:48:52.821552 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:48:53.326468 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:48:53.326430 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:49:03.350336 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:49:03.350299 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:49:03.452018 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:49:03.451984 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:49:03.517983 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:49:03.517951 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:49:07.566333 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:49:07.566294 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:49:07.663849 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:49:07.663818 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:49:34.201661 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:49:34.201573 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:49:34.301983 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:49:34.301945 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:49:57.637497 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:49:57.637355 2579 kubelet.go:1648] "Image garbage collection succeeded" Sep 12 01:50:00.837560 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:00.837530 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:50:02.318138 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:02.318100 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:50:04.482503 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:04.482445 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:50:04.582915 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:04.582882 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:50:06.279961 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:06.279922 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:50:06.376746 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:06.376710 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:50:06.913256 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:06.913222 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:50:07.027286 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:07.027251 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:50:08.304826 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:08.304785 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:50:08.685096 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:08.685066 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:50:11.443244 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:11.443201 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:50:11.507847 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:11.507815 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:50:17.096633 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:17.096582 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:50:18.021769 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:18.021740 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-rp8g5"] Sep 12 01:50:18.024965 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:18.024947 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-rp8g5" Sep 12 01:50:18.027433 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:18.027409 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 12 01:50:18.027585 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:18.027441 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 12 01:50:18.028301 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:18.028286 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-hqw8s\"" Sep 12 01:50:18.032620 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:18.032600 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-rp8g5"] Sep 12 01:50:18.116726 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:18.116681 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc5z9\" (UniqueName: \"kubernetes.io/projected/20773016-d768-4573-9aef-8da247687293-kube-api-access-mc5z9\") pod \"cert-manager-55b4d954d5-rp8g5\" (UID: \"20773016-d768-4573-9aef-8da247687293\") " pod="cert-manager/cert-manager-55b4d954d5-rp8g5" Sep 12 01:50:18.117101 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:18.116741 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/20773016-d768-4573-9aef-8da247687293-bound-sa-token\") pod \"cert-manager-55b4d954d5-rp8g5\" (UID: \"20773016-d768-4573-9aef-8da247687293\") " pod="cert-manager/cert-manager-55b4d954d5-rp8g5" Sep 12 01:50:18.217810 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:18.217769 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc5z9\" (UniqueName: \"kubernetes.io/projected/20773016-d768-4573-9aef-8da247687293-kube-api-access-mc5z9\") pod \"cert-manager-55b4d954d5-rp8g5\" (UID: \"20773016-d768-4573-9aef-8da247687293\") " pod="cert-manager/cert-manager-55b4d954d5-rp8g5" Sep 12 01:50:18.218012 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:18.217824 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/20773016-d768-4573-9aef-8da247687293-bound-sa-token\") pod \"cert-manager-55b4d954d5-rp8g5\" (UID: \"20773016-d768-4573-9aef-8da247687293\") " pod="cert-manager/cert-manager-55b4d954d5-rp8g5" Sep 12 01:50:18.226163 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:18.226126 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/20773016-d768-4573-9aef-8da247687293-bound-sa-token\") pod \"cert-manager-55b4d954d5-rp8g5\" (UID: \"20773016-d768-4573-9aef-8da247687293\") " pod="cert-manager/cert-manager-55b4d954d5-rp8g5" Sep 12 01:50:18.226330 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:18.226312 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc5z9\" (UniqueName: \"kubernetes.io/projected/20773016-d768-4573-9aef-8da247687293-kube-api-access-mc5z9\") pod \"cert-manager-55b4d954d5-rp8g5\" (UID: \"20773016-d768-4573-9aef-8da247687293\") " pod="cert-manager/cert-manager-55b4d954d5-rp8g5" Sep 12 01:50:18.351317 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:18.351230 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-rp8g5" Sep 12 01:50:18.628539 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:18.624622 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-rp8g5"] Sep 12 01:50:18.632730 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:50:18.632702 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod20773016_d768_4573_9aef_8da247687293.slice/crio-b646064d977d813088deb6ae8ba052e312822c4ec3876aa26bed4006ef26bd3f WatchSource:0}: Error finding container b646064d977d813088deb6ae8ba052e312822c4ec3876aa26bed4006ef26bd3f: Status 404 returned error can't find the container with id b646064d977d813088deb6ae8ba052e312822c4ec3876aa26bed4006ef26bd3f Sep 12 01:50:18.634620 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:18.634602 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 12 01:50:19.282035 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:19.281998 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-rp8g5" event={"ID":"20773016-d768-4573-9aef-8da247687293","Type":"ContainerStarted","Data":"b646064d977d813088deb6ae8ba052e312822c4ec3876aa26bed4006ef26bd3f"} Sep 12 01:50:22.296792 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:22.296755 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-rp8g5" event={"ID":"20773016-d768-4573-9aef-8da247687293","Type":"ContainerStarted","Data":"dff9e86c5274fbbad605104b5a23025cc8f1c3a95235642291c1dc927b04b83c"} Sep 12 01:50:22.312478 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:22.312399 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-rp8g5" podStartSLOduration=1.260259812 podStartE2EDuration="4.312378878s" totalImagesPullingTime="3.052119066s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:50:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:50:22.312004616 +0000 UTC m=+325.103998412" watchObservedRunningTime="2026-09-12 01:50:22.312378878 +0000 UTC m=+325.104372673" Sep 12 01:50:23.048535 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:23.048504 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:50:23.149948 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:23.149917 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:50:29.521681 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:29.521642 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:50:29.619581 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:29.619552 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:50:31.927428 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:31.927395 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:50:41.222134 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:41.222097 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:50:41.323141 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:41.323111 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:50:43.963595 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:43.963560 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:50:44.062236 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:44.062204 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:50:44.638030 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:44.637993 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:50:44.837260 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:44.837225 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:50:44.939141 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:44.939038 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:50:44.990036 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:44.990002 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-578c78994-v7hw8"] Sep 12 01:50:44.993686 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:44.993662 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-578c78994-v7hw8" Sep 12 01:50:44.997114 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:44.997093 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 12 01:50:44.997402 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:44.997383 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 12 01:50:44.997552 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:44.997447 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-qhcs5\"" Sep 12 01:50:44.997552 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:44.997481 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 12 01:50:44.997552 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:44.997510 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 12 01:50:45.027890 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:45.027858 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-578c78994-v7hw8"] Sep 12 01:50:45.048842 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:45.048810 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7hs9\" (UniqueName: \"kubernetes.io/projected/e61c2cac-5cce-4334-b88d-3677d05fb43a-kube-api-access-g7hs9\") pod \"opendatahub-operator-controller-manager-578c78994-v7hw8\" (UID: \"e61c2cac-5cce-4334-b88d-3677d05fb43a\") " pod="opendatahub/opendatahub-operator-controller-manager-578c78994-v7hw8" Sep 12 01:50:45.049019 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:45.048851 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e61c2cac-5cce-4334-b88d-3677d05fb43a-webhook-cert\") pod \"opendatahub-operator-controller-manager-578c78994-v7hw8\" (UID: \"e61c2cac-5cce-4334-b88d-3677d05fb43a\") " pod="opendatahub/opendatahub-operator-controller-manager-578c78994-v7hw8" Sep 12 01:50:45.049019 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:45.048983 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e61c2cac-5cce-4334-b88d-3677d05fb43a-manifests\") pod \"opendatahub-operator-controller-manager-578c78994-v7hw8\" (UID: \"e61c2cac-5cce-4334-b88d-3677d05fb43a\") " pod="opendatahub/opendatahub-operator-controller-manager-578c78994-v7hw8" Sep 12 01:50:45.049142 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:45.049016 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e61c2cac-5cce-4334-b88d-3677d05fb43a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-578c78994-v7hw8\" (UID: \"e61c2cac-5cce-4334-b88d-3677d05fb43a\") " pod="opendatahub/opendatahub-operator-controller-manager-578c78994-v7hw8" Sep 12 01:50:45.049142 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:45.049072 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e61c2cac-5cce-4334-b88d-3677d05fb43a-tmp\") pod \"opendatahub-operator-controller-manager-578c78994-v7hw8\" (UID: \"e61c2cac-5cce-4334-b88d-3677d05fb43a\") " pod="opendatahub/opendatahub-operator-controller-manager-578c78994-v7hw8" Sep 12 01:50:45.125606 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:45.125565 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:50:45.150522 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:45.150499 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e61c2cac-5cce-4334-b88d-3677d05fb43a-manifests\") pod \"opendatahub-operator-controller-manager-578c78994-v7hw8\" (UID: \"e61c2cac-5cce-4334-b88d-3677d05fb43a\") " pod="opendatahub/opendatahub-operator-controller-manager-578c78994-v7hw8" Sep 12 01:50:45.150656 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:45.150526 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e61c2cac-5cce-4334-b88d-3677d05fb43a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-578c78994-v7hw8\" (UID: \"e61c2cac-5cce-4334-b88d-3677d05fb43a\") " pod="opendatahub/opendatahub-operator-controller-manager-578c78994-v7hw8" Sep 12 01:50:45.150656 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:45.150552 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e61c2cac-5cce-4334-b88d-3677d05fb43a-tmp\") pod \"opendatahub-operator-controller-manager-578c78994-v7hw8\" (UID: \"e61c2cac-5cce-4334-b88d-3677d05fb43a\") " pod="opendatahub/opendatahub-operator-controller-manager-578c78994-v7hw8" Sep 12 01:50:45.150656 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:45.150584 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7hs9\" (UniqueName: \"kubernetes.io/projected/e61c2cac-5cce-4334-b88d-3677d05fb43a-kube-api-access-g7hs9\") pod \"opendatahub-operator-controller-manager-578c78994-v7hw8\" (UID: \"e61c2cac-5cce-4334-b88d-3677d05fb43a\") " pod="opendatahub/opendatahub-operator-controller-manager-578c78994-v7hw8" Sep 12 01:50:45.150656 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:45.150606 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e61c2cac-5cce-4334-b88d-3677d05fb43a-webhook-cert\") pod \"opendatahub-operator-controller-manager-578c78994-v7hw8\" (UID: \"e61c2cac-5cce-4334-b88d-3677d05fb43a\") " pod="opendatahub/opendatahub-operator-controller-manager-578c78994-v7hw8" Sep 12 01:50:45.150935 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:45.150911 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e61c2cac-5cce-4334-b88d-3677d05fb43a-manifests\") pod \"opendatahub-operator-controller-manager-578c78994-v7hw8\" (UID: \"e61c2cac-5cce-4334-b88d-3677d05fb43a\") " pod="opendatahub/opendatahub-operator-controller-manager-578c78994-v7hw8" Sep 12 01:50:45.153072 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:45.153044 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e61c2cac-5cce-4334-b88d-3677d05fb43a-tmp\") pod \"opendatahub-operator-controller-manager-578c78994-v7hw8\" (UID: \"e61c2cac-5cce-4334-b88d-3677d05fb43a\") " pod="opendatahub/opendatahub-operator-controller-manager-578c78994-v7hw8" Sep 12 01:50:45.153185 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:45.153169 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e61c2cac-5cce-4334-b88d-3677d05fb43a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-578c78994-v7hw8\" (UID: \"e61c2cac-5cce-4334-b88d-3677d05fb43a\") " pod="opendatahub/opendatahub-operator-controller-manager-578c78994-v7hw8" Sep 12 01:50:45.153236 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:45.153219 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e61c2cac-5cce-4334-b88d-3677d05fb43a-webhook-cert\") pod \"opendatahub-operator-controller-manager-578c78994-v7hw8\" (UID: \"e61c2cac-5cce-4334-b88d-3677d05fb43a\") " pod="opendatahub/opendatahub-operator-controller-manager-578c78994-v7hw8" Sep 12 01:50:45.158898 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:45.158879 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7hs9\" (UniqueName: \"kubernetes.io/projected/e61c2cac-5cce-4334-b88d-3677d05fb43a-kube-api-access-g7hs9\") pod \"opendatahub-operator-controller-manager-578c78994-v7hw8\" (UID: \"e61c2cac-5cce-4334-b88d-3677d05fb43a\") " pod="opendatahub/opendatahub-operator-controller-manager-578c78994-v7hw8" Sep 12 01:50:45.223858 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:45.223772 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:50:45.304447 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:45.304410 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-578c78994-v7hw8" Sep 12 01:50:45.338354 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:45.338326 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:50:45.432908 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:45.432871 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:50:45.533902 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:45.533814 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:50:45.598233 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:45.598207 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-578c78994-v7hw8"] Sep 12 01:50:45.600735 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:45.600708 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:50:45.602827 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:50:45.602795 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode61c2cac_5cce_4334_b88d_3677d05fb43a.slice/crio-3061aacf31994f8cd081aa68d75f7a517bfc4157b52a1eed424b8fb2caf0fe88 WatchSource:0}: Error finding container 3061aacf31994f8cd081aa68d75f7a517bfc4157b52a1eed424b8fb2caf0fe88: Status 404 returned error can't find the container with id 3061aacf31994f8cd081aa68d75f7a517bfc4157b52a1eed424b8fb2caf0fe88 Sep 12 01:50:46.191406 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:46.191371 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:50:46.291150 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:46.291113 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:50:46.398911 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:46.398869 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-578c78994-v7hw8" event={"ID":"e61c2cac-5cce-4334-b88d-3677d05fb43a","Type":"ContainerStarted","Data":"3061aacf31994f8cd081aa68d75f7a517bfc4157b52a1eed424b8fb2caf0fe88"} Sep 12 01:50:49.416941 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:49.416904 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-578c78994-v7hw8" event={"ID":"e61c2cac-5cce-4334-b88d-3677d05fb43a","Type":"ContainerStarted","Data":"5cfb3c5ab1dc895201404a0b0b007fc91ed6fe00a742f3e2276a3503fbe0996e"} Sep 12 01:50:50.422808 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:50.422772 2579 generic.go:356] "Generic (PLEG): container finished" podID="e61c2cac-5cce-4334-b88d-3677d05fb43a" containerID="5cfb3c5ab1dc895201404a0b0b007fc91ed6fe00a742f3e2276a3503fbe0996e" exitCode=0 Sep 12 01:50:50.423196 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:50.422868 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-578c78994-v7hw8" event={"ID":"e61c2cac-5cce-4334-b88d-3677d05fb43a","Type":"ContainerDied","Data":"5cfb3c5ab1dc895201404a0b0b007fc91ed6fe00a742f3e2276a3503fbe0996e"} Sep 12 01:50:52.432959 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:52.432925 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-578c78994-v7hw8" event={"ID":"e61c2cac-5cce-4334-b88d-3677d05fb43a","Type":"ContainerStarted","Data":"a8112df14926f719c04e6ec160abf5bfaaa10b27e475dfae5e33ff035f7ea36c"} Sep 12 01:50:52.433355 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:52.433002 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-578c78994-v7hw8" Sep 12 01:50:52.458143 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:52.458096 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-578c78994-v7hw8" podStartSLOduration=3.12967676 podStartE2EDuration="8.458080924s" totalImagesPullingTime="4.312129925s" totalInitContainerRuntime="1.016274239s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:50:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:50:52.455215232 +0000 UTC m=+355.247209030" watchObservedRunningTime="2026-09-12 01:50:52.458080924 +0000 UTC m=+355.250074718" Sep 12 01:50:55.624075 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:55.624040 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:50:55.723142 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:50:55.723107 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:51:03.440708 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:03.440681 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-578c78994-v7hw8" Sep 12 01:51:04.742241 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:04.742200 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-b9f9494c8-kjlfm"] Sep 12 01:51:04.750473 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:04.750424 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-b9f9494c8-kjlfm" Sep 12 01:51:04.753185 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:04.753158 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 12 01:51:04.753185 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:04.753160 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 12 01:51:04.754395 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:04.754382 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-7rmgc\"" Sep 12 01:51:04.754503 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:04.754485 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 12 01:51:04.754554 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:04.754516 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 12 01:51:04.754598 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:04.754517 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 12 01:51:04.760125 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:04.760104 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-b9f9494c8-kjlfm"] Sep 12 01:51:04.924187 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:04.924152 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ae104ce1-96f0-4774-bb2d-e17e478b4afd-manager-config\") pod \"lws-controller-manager-b9f9494c8-kjlfm\" (UID: \"ae104ce1-96f0-4774-bb2d-e17e478b4afd\") " pod="openshift-lws-operator/lws-controller-manager-b9f9494c8-kjlfm" Sep 12 01:51:04.924364 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:04.924225 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ae104ce1-96f0-4774-bb2d-e17e478b4afd-metrics-cert\") pod \"lws-controller-manager-b9f9494c8-kjlfm\" (UID: \"ae104ce1-96f0-4774-bb2d-e17e478b4afd\") " pod="openshift-lws-operator/lws-controller-manager-b9f9494c8-kjlfm" Sep 12 01:51:04.924364 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:04.924250 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c8gq\" (UniqueName: \"kubernetes.io/projected/ae104ce1-96f0-4774-bb2d-e17e478b4afd-kube-api-access-9c8gq\") pod \"lws-controller-manager-b9f9494c8-kjlfm\" (UID: \"ae104ce1-96f0-4774-bb2d-e17e478b4afd\") " pod="openshift-lws-operator/lws-controller-manager-b9f9494c8-kjlfm" Sep 12 01:51:04.924364 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:04.924306 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ae104ce1-96f0-4774-bb2d-e17e478b4afd-cert\") pod \"lws-controller-manager-b9f9494c8-kjlfm\" (UID: \"ae104ce1-96f0-4774-bb2d-e17e478b4afd\") " pod="openshift-lws-operator/lws-controller-manager-b9f9494c8-kjlfm" Sep 12 01:51:05.025775 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:05.025666 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ae104ce1-96f0-4774-bb2d-e17e478b4afd-manager-config\") pod \"lws-controller-manager-b9f9494c8-kjlfm\" (UID: \"ae104ce1-96f0-4774-bb2d-e17e478b4afd\") " pod="openshift-lws-operator/lws-controller-manager-b9f9494c8-kjlfm" Sep 12 01:51:05.025775 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:05.025734 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ae104ce1-96f0-4774-bb2d-e17e478b4afd-metrics-cert\") pod \"lws-controller-manager-b9f9494c8-kjlfm\" (UID: \"ae104ce1-96f0-4774-bb2d-e17e478b4afd\") " pod="openshift-lws-operator/lws-controller-manager-b9f9494c8-kjlfm" Sep 12 01:51:05.025979 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:05.025829 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c8gq\" (UniqueName: \"kubernetes.io/projected/ae104ce1-96f0-4774-bb2d-e17e478b4afd-kube-api-access-9c8gq\") pod \"lws-controller-manager-b9f9494c8-kjlfm\" (UID: \"ae104ce1-96f0-4774-bb2d-e17e478b4afd\") " pod="openshift-lws-operator/lws-controller-manager-b9f9494c8-kjlfm" Sep 12 01:51:05.025979 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:05.025932 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ae104ce1-96f0-4774-bb2d-e17e478b4afd-cert\") pod \"lws-controller-manager-b9f9494c8-kjlfm\" (UID: \"ae104ce1-96f0-4774-bb2d-e17e478b4afd\") " pod="openshift-lws-operator/lws-controller-manager-b9f9494c8-kjlfm" Sep 12 01:51:05.026321 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:05.026298 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ae104ce1-96f0-4774-bb2d-e17e478b4afd-manager-config\") pod \"lws-controller-manager-b9f9494c8-kjlfm\" (UID: \"ae104ce1-96f0-4774-bb2d-e17e478b4afd\") " pod="openshift-lws-operator/lws-controller-manager-b9f9494c8-kjlfm" Sep 12 01:51:05.028294 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:05.028264 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ae104ce1-96f0-4774-bb2d-e17e478b4afd-cert\") pod \"lws-controller-manager-b9f9494c8-kjlfm\" (UID: \"ae104ce1-96f0-4774-bb2d-e17e478b4afd\") " pod="openshift-lws-operator/lws-controller-manager-b9f9494c8-kjlfm" Sep 12 01:51:05.028669 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:05.028649 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ae104ce1-96f0-4774-bb2d-e17e478b4afd-metrics-cert\") pod \"lws-controller-manager-b9f9494c8-kjlfm\" (UID: \"ae104ce1-96f0-4774-bb2d-e17e478b4afd\") " pod="openshift-lws-operator/lws-controller-manager-b9f9494c8-kjlfm" Sep 12 01:51:05.043667 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:05.043642 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c8gq\" (UniqueName: \"kubernetes.io/projected/ae104ce1-96f0-4774-bb2d-e17e478b4afd-kube-api-access-9c8gq\") pod \"lws-controller-manager-b9f9494c8-kjlfm\" (UID: \"ae104ce1-96f0-4774-bb2d-e17e478b4afd\") " pod="openshift-lws-operator/lws-controller-manager-b9f9494c8-kjlfm" Sep 12 01:51:05.060303 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:05.060272 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-b9f9494c8-kjlfm" Sep 12 01:51:05.138854 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:05.138759 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:51:05.239382 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:05.239347 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:51:05.346334 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:05.346311 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-b9f9494c8-kjlfm"] Sep 12 01:51:05.349432 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:51:05.349405 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae104ce1_96f0_4774_bb2d_e17e478b4afd.slice/crio-2da9f93c451d0986a54eb77adecf3695073b4b2934976ac3195613736bd79fc5 WatchSource:0}: Error finding container 2da9f93c451d0986a54eb77adecf3695073b4b2934976ac3195613736bd79fc5: Status 404 returned error can't find the container with id 2da9f93c451d0986a54eb77adecf3695073b4b2934976ac3195613736bd79fc5 Sep 12 01:51:05.403866 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:05.403832 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:51:05.445339 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:05.445295 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-69c99bdc57-jdfsk"] Sep 12 01:51:05.448252 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:05.448235 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-69c99bdc57-jdfsk" Sep 12 01:51:05.450930 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:05.450909 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 12 01:51:05.451038 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:05.450934 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 12 01:51:05.451093 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:05.451034 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-q5vdz\"" Sep 12 01:51:05.451871 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:05.451843 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 12 01:51:05.451945 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:05.451874 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 12 01:51:05.461767 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:05.461741 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-69c99bdc57-jdfsk"] Sep 12 01:51:05.486731 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:05.486696 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:51:05.493361 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:05.493339 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-b9f9494c8-kjlfm" event={"ID":"ae104ce1-96f0-4774-bb2d-e17e478b4afd","Type":"ContainerStarted","Data":"2da9f93c451d0986a54eb77adecf3695073b4b2934976ac3195613736bd79fc5"} Sep 12 01:51:05.531818 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:05.531785 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53310321-9c0d-4246-9ed7-7ac33f3754b3-tls-certs\") pod \"kube-auth-proxy-69c99bdc57-jdfsk\" (UID: \"53310321-9c0d-4246-9ed7-7ac33f3754b3\") " pod="openshift-ingress/kube-auth-proxy-69c99bdc57-jdfsk" Sep 12 01:51:05.531957 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:05.531836 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/53310321-9c0d-4246-9ed7-7ac33f3754b3-tmp\") pod \"kube-auth-proxy-69c99bdc57-jdfsk\" (UID: \"53310321-9c0d-4246-9ed7-7ac33f3754b3\") " pod="openshift-ingress/kube-auth-proxy-69c99bdc57-jdfsk" Sep 12 01:51:05.531957 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:05.531859 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxkhf\" (UniqueName: \"kubernetes.io/projected/53310321-9c0d-4246-9ed7-7ac33f3754b3-kube-api-access-rxkhf\") pod \"kube-auth-proxy-69c99bdc57-jdfsk\" (UID: \"53310321-9c0d-4246-9ed7-7ac33f3754b3\") " pod="openshift-ingress/kube-auth-proxy-69c99bdc57-jdfsk" Sep 12 01:51:05.633000 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:05.632971 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53310321-9c0d-4246-9ed7-7ac33f3754b3-tls-certs\") pod \"kube-auth-proxy-69c99bdc57-jdfsk\" (UID: \"53310321-9c0d-4246-9ed7-7ac33f3754b3\") " pod="openshift-ingress/kube-auth-proxy-69c99bdc57-jdfsk" Sep 12 01:51:05.633191 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:05.633034 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/53310321-9c0d-4246-9ed7-7ac33f3754b3-tmp\") pod \"kube-auth-proxy-69c99bdc57-jdfsk\" (UID: \"53310321-9c0d-4246-9ed7-7ac33f3754b3\") " pod="openshift-ingress/kube-auth-proxy-69c99bdc57-jdfsk" Sep 12 01:51:05.633191 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:05.633080 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxkhf\" (UniqueName: \"kubernetes.io/projected/53310321-9c0d-4246-9ed7-7ac33f3754b3-kube-api-access-rxkhf\") pod \"kube-auth-proxy-69c99bdc57-jdfsk\" (UID: \"53310321-9c0d-4246-9ed7-7ac33f3754b3\") " pod="openshift-ingress/kube-auth-proxy-69c99bdc57-jdfsk" Sep 12 01:51:05.635323 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:05.635296 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/53310321-9c0d-4246-9ed7-7ac33f3754b3-tmp\") pod \"kube-auth-proxy-69c99bdc57-jdfsk\" (UID: \"53310321-9c0d-4246-9ed7-7ac33f3754b3\") " pod="openshift-ingress/kube-auth-proxy-69c99bdc57-jdfsk" Sep 12 01:51:05.635519 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:05.635504 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53310321-9c0d-4246-9ed7-7ac33f3754b3-tls-certs\") pod \"kube-auth-proxy-69c99bdc57-jdfsk\" (UID: \"53310321-9c0d-4246-9ed7-7ac33f3754b3\") " pod="openshift-ingress/kube-auth-proxy-69c99bdc57-jdfsk" Sep 12 01:51:05.643836 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:05.643816 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxkhf\" (UniqueName: \"kubernetes.io/projected/53310321-9c0d-4246-9ed7-7ac33f3754b3-kube-api-access-rxkhf\") pod \"kube-auth-proxy-69c99bdc57-jdfsk\" (UID: \"53310321-9c0d-4246-9ed7-7ac33f3754b3\") " pod="openshift-ingress/kube-auth-proxy-69c99bdc57-jdfsk" Sep 12 01:51:05.758315 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:05.758265 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-69c99bdc57-jdfsk" Sep 12 01:51:06.028674 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:06.028607 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-69c99bdc57-jdfsk"] Sep 12 01:51:06.031538 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:51:06.031512 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53310321_9c0d_4246_9ed7_7ac33f3754b3.slice/crio-bc6ae9ede2b658b4d331e39eb8225123dc0c993356169aa93054372d17e87e8c WatchSource:0}: Error finding container bc6ae9ede2b658b4d331e39eb8225123dc0c993356169aa93054372d17e87e8c: Status 404 returned error can't find the container with id bc6ae9ede2b658b4d331e39eb8225123dc0c993356169aa93054372d17e87e8c Sep 12 01:51:06.501847 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:06.501758 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-69c99bdc57-jdfsk" event={"ID":"53310321-9c0d-4246-9ed7-7ac33f3754b3","Type":"ContainerStarted","Data":"bc6ae9ede2b658b4d331e39eb8225123dc0c993356169aa93054372d17e87e8c"} Sep 12 01:51:06.683699 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:06.683631 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:51:06.765687 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:06.765551 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:51:07.819647 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:07.819590 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:51:10.524539 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:10.524503 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-69c99bdc57-jdfsk" event={"ID":"53310321-9c0d-4246-9ed7-7ac33f3754b3","Type":"ContainerStarted","Data":"6331de7a552bfed9d09df8107a9c6ae392aa66df368051107c15629fe9a68e97"} Sep 12 01:51:10.525932 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:10.525906 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-b9f9494c8-kjlfm" event={"ID":"ae104ce1-96f0-4774-bb2d-e17e478b4afd","Type":"ContainerStarted","Data":"92ee94ec554e89197fd095372078d14bb87a74308e02c06f5870a5e30432b31d"} Sep 12 01:51:10.526022 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:10.526007 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-b9f9494c8-kjlfm" Sep 12 01:51:10.541366 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:10.541306 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-69c99bdc57-jdfsk" podStartSLOduration=1.963000614 podStartE2EDuration="5.541290881s" totalImagesPullingTime="3.578290267s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:51:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:51:10.539997141 +0000 UTC m=+373.331990937" watchObservedRunningTime="2026-09-12 01:51:10.541290881 +0000 UTC m=+373.333284676" Sep 12 01:51:10.557235 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:10.557188 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-b9f9494c8-kjlfm" podStartSLOduration=2.340301166 podStartE2EDuration="6.557170468s" totalImagesPullingTime="4.216869302s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:51:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:51:10.55591903 +0000 UTC m=+373.347912823" watchObservedRunningTime="2026-09-12 01:51:10.557170468 +0000 UTC m=+373.349164262" Sep 12 01:51:11.306758 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:11.306728 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:51:11.407292 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:11.407262 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:51:11.497733 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:11.497698 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:51:11.723067 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:11.723040 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:51:11.848071 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:11.848030 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:51:14.226588 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:14.226553 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:51:15.567675 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:15.567638 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:51:15.667909 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:15.667850 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:51:15.771052 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:15.771017 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:51:16.144021 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:16.143988 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:51:16.246646 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:16.246602 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:51:16.344707 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:16.344675 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:51:16.447514 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:16.447425 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:51:16.534065 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:16.534026 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:51:16.765870 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:16.765789 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:51:21.532743 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:21.532710 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-b9f9494c8-kjlfm" Sep 12 01:51:25.605697 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:25.605660 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:51:25.709780 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:25.709747 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:51:25.946655 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:25.946619 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:51:26.098312 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:26.098281 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:51:26.186011 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:26.185980 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:51:26.287746 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:26.287661 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:51:26.883391 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:26.883354 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:51:26.962653 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:26.962622 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:51:27.098687 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:51:27.098653 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:52:17.388475 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:52:17.388416 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:52:17.455980 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:52:17.455935 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:52:27.229944 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:52:27.229850 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:52:27.305119 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:52:27.305085 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:52:29.075723 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:52:29.075681 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:52:41.584326 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:52:41.584287 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:52:44.368467 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:52:44.368426 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:52:45.971361 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:52:45.971328 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:52:46.399570 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:52:46.399532 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:52:46.632470 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:52:46.632414 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:52:48.040704 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:52:48.040665 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:52:49.006746 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:52:49.006707 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:52:49.106471 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:52:49.106430 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:52:49.364638 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:52:49.364545 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:52:49.468922 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:52:49.468887 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:52:49.759243 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:52:49.759213 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:52:49.870411 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:52:49.870366 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:52:50.502842 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:52:50.502809 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:52:54.208388 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:52:54.208348 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:52:54.310713 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:52:54.310680 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:52:57.429905 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:52:57.429873 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:52:57.641902 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:52:57.641866 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:53:01.209465 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:01.209424 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:53:01.390172 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:01.390137 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:53:02.453753 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:02.453722 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:53:02.596088 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:02.596055 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:53:03.218415 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:03.218378 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:53:03.367078 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:03.367049 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:53:05.716335 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:05.716294 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:53:05.896158 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:05.896125 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:53:09.365807 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:09.365770 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:53:09.444920 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:09.444873 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:53:09.795117 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:09.795086 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:53:09.935142 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:09.935108 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:53:10.057383 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:10.057298 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:53:10.224205 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:10.224157 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:53:11.765466 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:11.765418 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6srhb"] Sep 12 01:53:11.769147 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:11.769127 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6srhb" Sep 12 01:53:11.772414 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:11.772390 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 12 01:53:11.772581 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:11.772445 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-snrcf\"" Sep 12 01:53:11.772717 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:11.772703 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 12 01:53:11.788477 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:11.788428 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6srhb"] Sep 12 01:53:11.879506 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:11.879472 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v2bx\" (UniqueName: \"kubernetes.io/projected/958ef687-daa1-4bb8-9665-5ae669eaca49-kube-api-access-4v2bx\") pod \"kuadrant-operator-controller-manager-76f8f686bf-6srhb\" (UID: \"958ef687-daa1-4bb8-9665-5ae669eaca49\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6srhb" Sep 12 01:53:11.879670 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:11.879539 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/958ef687-daa1-4bb8-9665-5ae669eaca49-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-6srhb\" (UID: \"958ef687-daa1-4bb8-9665-5ae669eaca49\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6srhb" Sep 12 01:53:11.980322 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:11.980283 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v2bx\" (UniqueName: \"kubernetes.io/projected/958ef687-daa1-4bb8-9665-5ae669eaca49-kube-api-access-4v2bx\") pod \"kuadrant-operator-controller-manager-76f8f686bf-6srhb\" (UID: \"958ef687-daa1-4bb8-9665-5ae669eaca49\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6srhb" Sep 12 01:53:11.980559 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:11.980364 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/958ef687-daa1-4bb8-9665-5ae669eaca49-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-6srhb\" (UID: \"958ef687-daa1-4bb8-9665-5ae669eaca49\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6srhb" Sep 12 01:53:11.980836 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:11.980803 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/958ef687-daa1-4bb8-9665-5ae669eaca49-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-6srhb\" (UID: \"958ef687-daa1-4bb8-9665-5ae669eaca49\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6srhb" Sep 12 01:53:11.990672 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:11.990647 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v2bx\" (UniqueName: \"kubernetes.io/projected/958ef687-daa1-4bb8-9665-5ae669eaca49-kube-api-access-4v2bx\") pod \"kuadrant-operator-controller-manager-76f8f686bf-6srhb\" (UID: \"958ef687-daa1-4bb8-9665-5ae669eaca49\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6srhb" Sep 12 01:53:12.080967 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:12.080867 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6srhb" Sep 12 01:53:12.378959 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:12.378935 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6srhb"] Sep 12 01:53:12.381616 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:53:12.381591 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod958ef687_daa1_4bb8_9665_5ae669eaca49.slice/crio-998fd48f0ca9cb2015d5fcdfe5b9174b4855df8495ccaa19450c80c5ecb73bac WatchSource:0}: Error finding container 998fd48f0ca9cb2015d5fcdfe5b9174b4855df8495ccaa19450c80c5ecb73bac: Status 404 returned error can't find the container with id 998fd48f0ca9cb2015d5fcdfe5b9174b4855df8495ccaa19450c80c5ecb73bac Sep 12 01:53:13.090999 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:13.090965 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6srhb" event={"ID":"958ef687-daa1-4bb8-9665-5ae669eaca49","Type":"ContainerStarted","Data":"998fd48f0ca9cb2015d5fcdfe5b9174b4855df8495ccaa19450c80c5ecb73bac"} Sep 12 01:53:15.068314 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:15.068259 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:53:15.181204 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:15.181166 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:53:17.114269 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:17.114188 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6srhb" event={"ID":"958ef687-daa1-4bb8-9665-5ae669eaca49","Type":"ContainerStarted","Data":"8d4beedba722f3faf1b5262b9c8de682fd5b71e46819875d02b8aaa730074f7c"} Sep 12 01:53:17.114623 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:17.114271 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6srhb" Sep 12 01:53:17.142383 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:17.142335 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6srhb" podStartSLOduration=1.781417507 podStartE2EDuration="6.142319923s" totalImagesPullingTime="4.360902416s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:53:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:53:17.141641542 +0000 UTC m=+499.933635338" watchObservedRunningTime="2026-09-12 01:53:17.142319923 +0000 UTC m=+499.934313718" Sep 12 01:53:18.257483 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:18.257425 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:53:18.355929 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:18.355894 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:53:22.702744 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:22.702703 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:53:22.774664 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:22.774629 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:53:22.874064 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:22.874028 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:53:22.957411 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:22.957324 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:53:27.572600 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:27.572563 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6srhb"] Sep 12 01:53:27.573017 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:27.572799 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6srhb" podUID="958ef687-daa1-4bb8-9665-5ae669eaca49" containerName="manager" containerID="cri-o://8d4beedba722f3faf1b5262b9c8de682fd5b71e46819875d02b8aaa730074f7c" gracePeriod=10 Sep 12 01:53:27.574697 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:27.574678 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6srhb" Sep 12 01:53:27.819166 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:27.819143 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6srhb" Sep 12 01:53:27.881643 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:27.881607 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:53:27.928199 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:27.928148 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/958ef687-daa1-4bb8-9665-5ae669eaca49-extensions-socket-volume\") pod \"958ef687-daa1-4bb8-9665-5ae669eaca49\" (UID: \"958ef687-daa1-4bb8-9665-5ae669eaca49\") " Sep 12 01:53:27.928358 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:27.928235 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4v2bx\" (UniqueName: \"kubernetes.io/projected/958ef687-daa1-4bb8-9665-5ae669eaca49-kube-api-access-4v2bx\") pod \"958ef687-daa1-4bb8-9665-5ae669eaca49\" (UID: \"958ef687-daa1-4bb8-9665-5ae669eaca49\") " Sep 12 01:53:27.928765 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:27.928738 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/958ef687-daa1-4bb8-9665-5ae669eaca49-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "958ef687-daa1-4bb8-9665-5ae669eaca49" (UID: "958ef687-daa1-4bb8-9665-5ae669eaca49"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 12 01:53:27.930528 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:27.930509 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/958ef687-daa1-4bb8-9665-5ae669eaca49-kube-api-access-4v2bx" (OuterVolumeSpecName: "kube-api-access-4v2bx") pod "958ef687-daa1-4bb8-9665-5ae669eaca49" (UID: "958ef687-daa1-4bb8-9665-5ae669eaca49"). InnerVolumeSpecName "kube-api-access-4v2bx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 12 01:53:27.987677 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:27.987639 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:53:28.029160 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:28.029135 2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/958ef687-daa1-4bb8-9665-5ae669eaca49-extensions-socket-volume\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 01:53:28.029160 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:28.029160 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4v2bx\" (UniqueName: \"kubernetes.io/projected/958ef687-daa1-4bb8-9665-5ae669eaca49-kube-api-access-4v2bx\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 01:53:28.087719 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:28.087679 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:53:28.174816 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:28.174717 2579 generic.go:356] "Generic (PLEG): container finished" podID="958ef687-daa1-4bb8-9665-5ae669eaca49" containerID="8d4beedba722f3faf1b5262b9c8de682fd5b71e46819875d02b8aaa730074f7c" exitCode=0 Sep 12 01:53:28.174816 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:28.174770 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6srhb" Sep 12 01:53:28.174816 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:28.174786 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6srhb" event={"ID":"958ef687-daa1-4bb8-9665-5ae669eaca49","Type":"ContainerDied","Data":"8d4beedba722f3faf1b5262b9c8de682fd5b71e46819875d02b8aaa730074f7c"} Sep 12 01:53:28.175086 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:28.174859 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6srhb" event={"ID":"958ef687-daa1-4bb8-9665-5ae669eaca49","Type":"ContainerDied","Data":"998fd48f0ca9cb2015d5fcdfe5b9174b4855df8495ccaa19450c80c5ecb73bac"} Sep 12 01:53:28.175086 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:28.174887 2579 scope.go:117] "RemoveContainer" containerID="8d4beedba722f3faf1b5262b9c8de682fd5b71e46819875d02b8aaa730074f7c" Sep 12 01:53:28.183807 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:28.183788 2579 scope.go:117] "RemoveContainer" containerID="8d4beedba722f3faf1b5262b9c8de682fd5b71e46819875d02b8aaa730074f7c" Sep 12 01:53:28.184095 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:53:28.184077 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d4beedba722f3faf1b5262b9c8de682fd5b71e46819875d02b8aaa730074f7c\": container with ID starting with 8d4beedba722f3faf1b5262b9c8de682fd5b71e46819875d02b8aaa730074f7c not found: ID does not exist" containerID="8d4beedba722f3faf1b5262b9c8de682fd5b71e46819875d02b8aaa730074f7c" Sep 12 01:53:28.184162 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:28.184104 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d4beedba722f3faf1b5262b9c8de682fd5b71e46819875d02b8aaa730074f7c"} err="failed to get container status \"8d4beedba722f3faf1b5262b9c8de682fd5b71e46819875d02b8aaa730074f7c\": rpc error: code = NotFound desc = could not find container \"8d4beedba722f3faf1b5262b9c8de682fd5b71e46819875d02b8aaa730074f7c\": container with ID starting with 8d4beedba722f3faf1b5262b9c8de682fd5b71e46819875d02b8aaa730074f7c not found: ID does not exist" Sep 12 01:53:28.205170 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:28.205144 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6srhb"] Sep 12 01:53:28.209743 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:28.209716 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6srhb"] Sep 12 01:53:29.726259 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:29.726227 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="958ef687-daa1-4bb8-9665-5ae669eaca49" path="/var/lib/kubelet/pods/958ef687-daa1-4bb8-9665-5ae669eaca49/volumes" Sep 12 01:53:42.124644 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:42.124608 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:53:44.037108 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:44.037076 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:53:44.253856 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:44.253816 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:53:49.447144 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:49.447066 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:53:51.573301 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:51.573263 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:53:56.010097 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:56.010048 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:53:56.129868 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:56.129837 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:53:56.230348 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:56.230309 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:53:56.430802 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:56.430773 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 01:53:56.431143 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:56.431131 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="958ef687-daa1-4bb8-9665-5ae669eaca49" containerName="manager" Sep 12 01:53:56.431196 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:56.431144 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="958ef687-daa1-4bb8-9665-5ae669eaca49" containerName="manager" Sep 12 01:53:56.431245 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:56.431206 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="958ef687-daa1-4bb8-9665-5ae669eaca49" containerName="manager" Sep 12 01:53:56.433079 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:56.433063 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-p5xqc" Sep 12 01:53:56.435346 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:56.435327 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 12 01:53:56.435507 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:56.435487 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 12 01:53:56.435583 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:56.435566 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 12 01:53:56.436102 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:56.436087 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-bf5qb\"" Sep 12 01:53:56.442366 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:56.442344 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 01:53:56.477150 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:56.477121 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 01:53:56.493114 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:56.493086 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a596d221-6246-4003-972e-6f658b315220-config-file\") pod \"limitador-limitador-84994f7ffc-p5xqc\" (UID: \"a596d221-6246-4003-972e-6f658b315220\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-p5xqc" Sep 12 01:53:56.493258 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:56.493144 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vw4m\" (UniqueName: \"kubernetes.io/projected/a596d221-6246-4003-972e-6f658b315220-kube-api-access-8vw4m\") pod \"limitador-limitador-84994f7ffc-p5xqc\" (UID: \"a596d221-6246-4003-972e-6f658b315220\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-p5xqc" Sep 12 01:53:56.594672 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:56.594634 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a596d221-6246-4003-972e-6f658b315220-config-file\") pod \"limitador-limitador-84994f7ffc-p5xqc\" (UID: \"a596d221-6246-4003-972e-6f658b315220\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-p5xqc" Sep 12 01:53:56.594865 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:56.594695 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vw4m\" (UniqueName: \"kubernetes.io/projected/a596d221-6246-4003-972e-6f658b315220-kube-api-access-8vw4m\") pod \"limitador-limitador-84994f7ffc-p5xqc\" (UID: \"a596d221-6246-4003-972e-6f658b315220\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-p5xqc" Sep 12 01:53:56.595290 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:56.595270 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a596d221-6246-4003-972e-6f658b315220-config-file\") pod \"limitador-limitador-84994f7ffc-p5xqc\" (UID: \"a596d221-6246-4003-972e-6f658b315220\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-p5xqc" Sep 12 01:53:56.604621 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:56.604599 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vw4m\" (UniqueName: \"kubernetes.io/projected/a596d221-6246-4003-972e-6f658b315220-kube-api-access-8vw4m\") pod \"limitador-limitador-84994f7ffc-p5xqc\" (UID: \"a596d221-6246-4003-972e-6f658b315220\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-p5xqc" Sep 12 01:53:56.745602 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:56.745510 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-p5xqc" Sep 12 01:53:56.849557 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:56.849524 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-9dkkv"] Sep 12 01:53:56.856799 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:56.856774 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-9dkkv" Sep 12 01:53:56.859241 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:56.859216 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-56m6m\"" Sep 12 01:53:56.861367 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:56.861342 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-9dkkv"] Sep 12 01:53:56.879989 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:56.879962 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:53:56.961089 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:56.961053 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:53:56.997767 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:56.997694 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvrdp\" (UniqueName: \"kubernetes.io/projected/d3a5c97b-7f88-4fc5-b8b1-d0e64112270c-kube-api-access-kvrdp\") pod \"authorino-7bcfd8f9cd-9dkkv\" (UID: \"d3a5c97b-7f88-4fc5-b8b1-d0e64112270c\") " pod="kuadrant-system/authorino-7bcfd8f9cd-9dkkv" Sep 12 01:53:57.013928 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:57.013894 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 01:53:57.017399 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:53:57.017367 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda596d221_6246_4003_972e_6f658b315220.slice/crio-c0c9fdc851861099340353ea21686e2bbbf9be51cdb1db9de5f213e629f927ad WatchSource:0}: Error finding container c0c9fdc851861099340353ea21686e2bbbf9be51cdb1db9de5f213e629f927ad: Status 404 returned error can't find the container with id c0c9fdc851861099340353ea21686e2bbbf9be51cdb1db9de5f213e629f927ad Sep 12 01:53:57.098660 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:57.098624 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvrdp\" (UniqueName: \"kubernetes.io/projected/d3a5c97b-7f88-4fc5-b8b1-d0e64112270c-kube-api-access-kvrdp\") pod \"authorino-7bcfd8f9cd-9dkkv\" (UID: \"d3a5c97b-7f88-4fc5-b8b1-d0e64112270c\") " pod="kuadrant-system/authorino-7bcfd8f9cd-9dkkv" Sep 12 01:53:57.108169 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:57.108149 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvrdp\" (UniqueName: \"kubernetes.io/projected/d3a5c97b-7f88-4fc5-b8b1-d0e64112270c-kube-api-access-kvrdp\") pod \"authorino-7bcfd8f9cd-9dkkv\" (UID: \"d3a5c97b-7f88-4fc5-b8b1-d0e64112270c\") " pod="kuadrant-system/authorino-7bcfd8f9cd-9dkkv" Sep 12 01:53:57.168737 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:57.168706 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-9dkkv" Sep 12 01:53:57.313395 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:57.313313 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-p5xqc" event={"ID":"a596d221-6246-4003-972e-6f658b315220","Type":"ContainerStarted","Data":"c0c9fdc851861099340353ea21686e2bbbf9be51cdb1db9de5f213e629f927ad"} Sep 12 01:53:57.437192 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:57.437165 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-9dkkv"] Sep 12 01:53:57.439780 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:53:57.439749 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3a5c97b_7f88_4fc5_b8b1_d0e64112270c.slice/crio-2941143e5604e0a8524e64ef210565579287c104c0ae39ae147ec9f010fe8adc WatchSource:0}: Error finding container 2941143e5604e0a8524e64ef210565579287c104c0ae39ae147ec9f010fe8adc: Status 404 returned error can't find the container with id 2941143e5604e0a8524e64ef210565579287c104c0ae39ae147ec9f010fe8adc Sep 12 01:53:58.320219 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:58.320125 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-p5xqc" event={"ID":"a596d221-6246-4003-972e-6f658b315220","Type":"ContainerStarted","Data":"8b0df7a55708da75cf4d74f5488e8d9bdfdb082e1c191591f9c4c460374a434c"} Sep 12 01:53:58.320644 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:58.320237 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-p5xqc" Sep 12 01:53:58.322012 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:58.321987 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-9dkkv" event={"ID":"d3a5c97b-7f88-4fc5-b8b1-d0e64112270c","Type":"ContainerStarted","Data":"2941143e5604e0a8524e64ef210565579287c104c0ae39ae147ec9f010fe8adc"} Sep 12 01:53:58.339403 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:58.339352 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-p5xqc" podStartSLOduration=1.442076399 podStartE2EDuration="2.339334434s" totalImagesPullingTime="897.258035ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:53:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:53:58.337029825 +0000 UTC m=+541.129023622" watchObservedRunningTime="2026-09-12 01:53:58.339334434 +0000 UTC m=+541.131328228" Sep 12 01:53:59.328745 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:59.328716 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-9dkkv" event={"ID":"d3a5c97b-7f88-4fc5-b8b1-d0e64112270c","Type":"ContainerStarted","Data":"6ee93a42b1d8f2b55e0a07848eed36f849a1640ec2a9e15eaafcec07c4f1415d"} Sep 12 01:53:59.345494 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:59.345430 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-9dkkv" podStartSLOduration=1.65641992 podStartE2EDuration="3.345416065s" totalImagesPullingTime="1.688996145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:53:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:53:59.344220244 +0000 UTC m=+542.136214042" watchObservedRunningTime="2026-09-12 01:53:59.345416065 +0000 UTC m=+542.137409860" Sep 12 01:53:59.745585 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:53:59.745547 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-9dkkv"] Sep 12 01:54:01.337614 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:01.337577 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-9dkkv" podUID="d3a5c97b-7f88-4fc5-b8b1-d0e64112270c" containerName="authorino" containerID="cri-o://6ee93a42b1d8f2b55e0a07848eed36f849a1640ec2a9e15eaafcec07c4f1415d" gracePeriod=30 Sep 12 01:54:01.574539 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:01.574517 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-9dkkv" Sep 12 01:54:01.640766 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:01.640728 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvrdp\" (UniqueName: \"kubernetes.io/projected/d3a5c97b-7f88-4fc5-b8b1-d0e64112270c-kube-api-access-kvrdp\") pod \"d3a5c97b-7f88-4fc5-b8b1-d0e64112270c\" (UID: \"d3a5c97b-7f88-4fc5-b8b1-d0e64112270c\") " Sep 12 01:54:01.642891 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:01.642865 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3a5c97b-7f88-4fc5-b8b1-d0e64112270c-kube-api-access-kvrdp" (OuterVolumeSpecName: "kube-api-access-kvrdp") pod "d3a5c97b-7f88-4fc5-b8b1-d0e64112270c" (UID: "d3a5c97b-7f88-4fc5-b8b1-d0e64112270c"). InnerVolumeSpecName "kube-api-access-kvrdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 12 01:54:01.741596 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:01.741568 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvrdp\" (UniqueName: \"kubernetes.io/projected/d3a5c97b-7f88-4fc5-b8b1-d0e64112270c-kube-api-access-kvrdp\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 01:54:02.343896 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:02.343861 2579 generic.go:356] "Generic (PLEG): container finished" podID="d3a5c97b-7f88-4fc5-b8b1-d0e64112270c" containerID="6ee93a42b1d8f2b55e0a07848eed36f849a1640ec2a9e15eaafcec07c4f1415d" exitCode=0 Sep 12 01:54:02.344327 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:02.343902 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-9dkkv" Sep 12 01:54:02.344327 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:02.343900 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-9dkkv" event={"ID":"d3a5c97b-7f88-4fc5-b8b1-d0e64112270c","Type":"ContainerDied","Data":"6ee93a42b1d8f2b55e0a07848eed36f849a1640ec2a9e15eaafcec07c4f1415d"} Sep 12 01:54:02.344327 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:02.344030 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-9dkkv" event={"ID":"d3a5c97b-7f88-4fc5-b8b1-d0e64112270c","Type":"ContainerDied","Data":"2941143e5604e0a8524e64ef210565579287c104c0ae39ae147ec9f010fe8adc"} Sep 12 01:54:02.344327 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:02.344052 2579 scope.go:117] "RemoveContainer" containerID="6ee93a42b1d8f2b55e0a07848eed36f849a1640ec2a9e15eaafcec07c4f1415d" Sep 12 01:54:02.352546 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:02.352530 2579 scope.go:117] "RemoveContainer" containerID="6ee93a42b1d8f2b55e0a07848eed36f849a1640ec2a9e15eaafcec07c4f1415d" Sep 12 01:54:02.352804 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:54:02.352782 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ee93a42b1d8f2b55e0a07848eed36f849a1640ec2a9e15eaafcec07c4f1415d\": container with ID starting with 6ee93a42b1d8f2b55e0a07848eed36f849a1640ec2a9e15eaafcec07c4f1415d not found: ID does not exist" containerID="6ee93a42b1d8f2b55e0a07848eed36f849a1640ec2a9e15eaafcec07c4f1415d" Sep 12 01:54:02.352864 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:02.352812 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ee93a42b1d8f2b55e0a07848eed36f849a1640ec2a9e15eaafcec07c4f1415d"} err="failed to get container status \"6ee93a42b1d8f2b55e0a07848eed36f849a1640ec2a9e15eaafcec07c4f1415d\": rpc error: code = NotFound desc = could not find container \"6ee93a42b1d8f2b55e0a07848eed36f849a1640ec2a9e15eaafcec07c4f1415d\": container with ID starting with 6ee93a42b1d8f2b55e0a07848eed36f849a1640ec2a9e15eaafcec07c4f1415d not found: ID does not exist" Sep 12 01:54:02.359466 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:02.359319 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-9dkkv"] Sep 12 01:54:02.361092 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:02.361071 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-9dkkv"] Sep 12 01:54:03.725983 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:03.725947 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3a5c97b-7f88-4fc5-b8b1-d0e64112270c" path="/var/lib/kubelet/pods/d3a5c97b-7f88-4fc5-b8b1-d0e64112270c/volumes" Sep 12 01:54:09.329700 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:09.329660 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-p5xqc" Sep 12 01:54:15.114856 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:15.114821 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:54:15.215065 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:15.215032 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:54:15.617223 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:15.617186 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:54:28.167399 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:28.167359 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:54:28.411804 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:28.411770 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-tl8np"] Sep 12 01:54:28.412132 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:28.412121 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3a5c97b-7f88-4fc5-b8b1-d0e64112270c" containerName="authorino" Sep 12 01:54:28.412177 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:28.412135 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3a5c97b-7f88-4fc5-b8b1-d0e64112270c" containerName="authorino" Sep 12 01:54:28.412229 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:28.412220 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3a5c97b-7f88-4fc5-b8b1-d0e64112270c" containerName="authorino" Sep 12 01:54:28.415342 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:28.415327 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-tl8np" Sep 12 01:54:28.417815 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:28.417752 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-56m6m\"" Sep 12 01:54:28.423011 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:28.422988 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-tl8np"] Sep 12 01:54:28.495302 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:28.495262 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxm2h\" (UniqueName: \"kubernetes.io/projected/c2a37575-b9fb-41ad-b080-d0d9b36f5866-kube-api-access-mxm2h\") pod \"authorino-557f876f69-tl8np\" (UID: \"c2a37575-b9fb-41ad-b080-d0d9b36f5866\") " pod="kuadrant-system/authorino-557f876f69-tl8np" Sep 12 01:54:28.556881 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:28.556845 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:54:28.596264 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:28.596220 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxm2h\" (UniqueName: \"kubernetes.io/projected/c2a37575-b9fb-41ad-b080-d0d9b36f5866-kube-api-access-mxm2h\") pod \"authorino-557f876f69-tl8np\" (UID: \"c2a37575-b9fb-41ad-b080-d0d9b36f5866\") " pod="kuadrant-system/authorino-557f876f69-tl8np" Sep 12 01:54:28.605389 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:28.605355 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxm2h\" (UniqueName: \"kubernetes.io/projected/c2a37575-b9fb-41ad-b080-d0d9b36f5866-kube-api-access-mxm2h\") pod \"authorino-557f876f69-tl8np\" (UID: \"c2a37575-b9fb-41ad-b080-d0d9b36f5866\") " pod="kuadrant-system/authorino-557f876f69-tl8np" Sep 12 01:54:28.647310 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:28.647276 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-tl8np"] Sep 12 01:54:28.647588 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:28.647575 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-tl8np" Sep 12 01:54:28.757095 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:28.757059 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6fc68b8b94-8gl2b"] Sep 12 01:54:28.760671 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:28.760656 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6fc68b8b94-8gl2b" Sep 12 01:54:28.763042 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:28.763019 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 12 01:54:28.768855 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:28.768828 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6fc68b8b94-8gl2b"] Sep 12 01:54:28.797643 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:28.797615 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fd8bf9f4-8be6-4918-8a22-511605c38365-tls-cert\") pod \"authorino-6fc68b8b94-8gl2b\" (UID: \"fd8bf9f4-8be6-4918-8a22-511605c38365\") " pod="kuadrant-system/authorino-6fc68b8b94-8gl2b" Sep 12 01:54:28.797815 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:28.797719 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68k7r\" (UniqueName: \"kubernetes.io/projected/fd8bf9f4-8be6-4918-8a22-511605c38365-kube-api-access-68k7r\") pod \"authorino-6fc68b8b94-8gl2b\" (UID: \"fd8bf9f4-8be6-4918-8a22-511605c38365\") " pod="kuadrant-system/authorino-6fc68b8b94-8gl2b" Sep 12 01:54:28.898423 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:28.898394 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68k7r\" (UniqueName: \"kubernetes.io/projected/fd8bf9f4-8be6-4918-8a22-511605c38365-kube-api-access-68k7r\") pod \"authorino-6fc68b8b94-8gl2b\" (UID: \"fd8bf9f4-8be6-4918-8a22-511605c38365\") " pod="kuadrant-system/authorino-6fc68b8b94-8gl2b" Sep 12 01:54:28.898612 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:28.898439 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fd8bf9f4-8be6-4918-8a22-511605c38365-tls-cert\") pod \"authorino-6fc68b8b94-8gl2b\" (UID: \"fd8bf9f4-8be6-4918-8a22-511605c38365\") " pod="kuadrant-system/authorino-6fc68b8b94-8gl2b" Sep 12 01:54:28.901027 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:28.900993 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fd8bf9f4-8be6-4918-8a22-511605c38365-tls-cert\") pod \"authorino-6fc68b8b94-8gl2b\" (UID: \"fd8bf9f4-8be6-4918-8a22-511605c38365\") " pod="kuadrant-system/authorino-6fc68b8b94-8gl2b" Sep 12 01:54:28.906742 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:28.906718 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68k7r\" (UniqueName: \"kubernetes.io/projected/fd8bf9f4-8be6-4918-8a22-511605c38365-kube-api-access-68k7r\") pod \"authorino-6fc68b8b94-8gl2b\" (UID: \"fd8bf9f4-8be6-4918-8a22-511605c38365\") " pod="kuadrant-system/authorino-6fc68b8b94-8gl2b" Sep 12 01:54:28.919417 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:28.919396 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-tl8np"] Sep 12 01:54:28.921568 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:54:28.921543 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc2a37575_b9fb_41ad_b080_d0d9b36f5866.slice/crio-38b2f63e151c041fc8fb8b96c1594f1dc1a60255d138e5fb73b5c86d11710345 WatchSource:0}: Error finding container 38b2f63e151c041fc8fb8b96c1594f1dc1a60255d138e5fb73b5c86d11710345: Status 404 returned error can't find the container with id 38b2f63e151c041fc8fb8b96c1594f1dc1a60255d138e5fb73b5c86d11710345 Sep 12 01:54:29.071329 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:29.071226 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6fc68b8b94-8gl2b" Sep 12 01:54:29.342570 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:29.342496 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6fc68b8b94-8gl2b"] Sep 12 01:54:29.345468 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:54:29.345427 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd8bf9f4_8be6_4918_8a22_511605c38365.slice/crio-7c596d6f1285b2d20a3a896e0ade99db98244c2ba59fe8a7f3efb6e613d937c2 WatchSource:0}: Error finding container 7c596d6f1285b2d20a3a896e0ade99db98244c2ba59fe8a7f3efb6e613d937c2: Status 404 returned error can't find the container with id 7c596d6f1285b2d20a3a896e0ade99db98244c2ba59fe8a7f3efb6e613d937c2 Sep 12 01:54:29.474906 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:29.474874 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-tl8np" event={"ID":"c2a37575-b9fb-41ad-b080-d0d9b36f5866","Type":"ContainerStarted","Data":"a850d75a364f4f052becb3e7f2f71819c013454e6da968bba1972148310151c3"} Sep 12 01:54:29.474906 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:29.474901 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-557f876f69-tl8np" podUID="c2a37575-b9fb-41ad-b080-d0d9b36f5866" containerName="authorino" containerID="cri-o://a850d75a364f4f052becb3e7f2f71819c013454e6da968bba1972148310151c3" gracePeriod=30 Sep 12 01:54:29.474906 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:29.474920 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-tl8np" event={"ID":"c2a37575-b9fb-41ad-b080-d0d9b36f5866","Type":"ContainerStarted","Data":"38b2f63e151c041fc8fb8b96c1594f1dc1a60255d138e5fb73b5c86d11710345"} Sep 12 01:54:29.476135 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:29.476105 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6fc68b8b94-8gl2b" event={"ID":"fd8bf9f4-8be6-4918-8a22-511605c38365","Type":"ContainerStarted","Data":"7c596d6f1285b2d20a3a896e0ade99db98244c2ba59fe8a7f3efb6e613d937c2"} Sep 12 01:54:29.489425 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:29.489381 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-557f876f69-tl8np" podStartSLOduration=1.044117034 podStartE2EDuration="1.489365669s" totalImagesPullingTime="445.248635ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:54:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:54:29.488601833 +0000 UTC m=+572.280595629" watchObservedRunningTime="2026-09-12 01:54:29.489365669 +0000 UTC m=+572.281359465" Sep 12 01:54:29.720502 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:29.720477 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-tl8np" Sep 12 01:54:29.806745 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:29.806709 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mxm2h\" (UniqueName: \"kubernetes.io/projected/c2a37575-b9fb-41ad-b080-d0d9b36f5866-kube-api-access-mxm2h\") pod \"c2a37575-b9fb-41ad-b080-d0d9b36f5866\" (UID: \"c2a37575-b9fb-41ad-b080-d0d9b36f5866\") " Sep 12 01:54:29.808933 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:29.808909 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2a37575-b9fb-41ad-b080-d0d9b36f5866-kube-api-access-mxm2h" (OuterVolumeSpecName: "kube-api-access-mxm2h") pod "c2a37575-b9fb-41ad-b080-d0d9b36f5866" (UID: "c2a37575-b9fb-41ad-b080-d0d9b36f5866"). InnerVolumeSpecName "kube-api-access-mxm2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 12 01:54:29.907681 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:29.907650 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxm2h\" (UniqueName: \"kubernetes.io/projected/c2a37575-b9fb-41ad-b080-d0d9b36f5866-kube-api-access-mxm2h\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 01:54:30.483649 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:30.483607 2579 generic.go:356] "Generic (PLEG): container finished" podID="c2a37575-b9fb-41ad-b080-d0d9b36f5866" containerID="a850d75a364f4f052becb3e7f2f71819c013454e6da968bba1972148310151c3" exitCode=0 Sep 12 01:54:30.484099 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:30.483667 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-tl8np" Sep 12 01:54:30.484099 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:30.483693 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-tl8np" event={"ID":"c2a37575-b9fb-41ad-b080-d0d9b36f5866","Type":"ContainerDied","Data":"a850d75a364f4f052becb3e7f2f71819c013454e6da968bba1972148310151c3"} Sep 12 01:54:30.484099 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:30.483732 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-tl8np" event={"ID":"c2a37575-b9fb-41ad-b080-d0d9b36f5866","Type":"ContainerDied","Data":"38b2f63e151c041fc8fb8b96c1594f1dc1a60255d138e5fb73b5c86d11710345"} Sep 12 01:54:30.484099 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:30.483753 2579 scope.go:117] "RemoveContainer" containerID="a850d75a364f4f052becb3e7f2f71819c013454e6da968bba1972148310151c3" Sep 12 01:54:30.485401 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:30.485371 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6fc68b8b94-8gl2b" event={"ID":"fd8bf9f4-8be6-4918-8a22-511605c38365","Type":"ContainerStarted","Data":"f23296ab11ca8aa833b61395a8817f5326767d0fe5ebde8bc9bba2fff73acd00"} Sep 12 01:54:30.492952 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:30.492928 2579 scope.go:117] "RemoveContainer" containerID="a850d75a364f4f052becb3e7f2f71819c013454e6da968bba1972148310151c3" Sep 12 01:54:30.493223 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:54:30.493204 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a850d75a364f4f052becb3e7f2f71819c013454e6da968bba1972148310151c3\": container with ID starting with a850d75a364f4f052becb3e7f2f71819c013454e6da968bba1972148310151c3 not found: ID does not exist" containerID="a850d75a364f4f052becb3e7f2f71819c013454e6da968bba1972148310151c3" Sep 12 01:54:30.493296 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:30.493235 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a850d75a364f4f052becb3e7f2f71819c013454e6da968bba1972148310151c3"} err="failed to get container status \"a850d75a364f4f052becb3e7f2f71819c013454e6da968bba1972148310151c3\": rpc error: code = NotFound desc = could not find container \"a850d75a364f4f052becb3e7f2f71819c013454e6da968bba1972148310151c3\": container with ID starting with a850d75a364f4f052becb3e7f2f71819c013454e6da968bba1972148310151c3 not found: ID does not exist" Sep 12 01:54:30.503538 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:30.503491 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6fc68b8b94-8gl2b" podStartSLOduration=1.889182066 podStartE2EDuration="2.503476378s" totalImagesPullingTime="614.294312ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:54:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:54:30.502224197 +0000 UTC m=+573.294217994" watchObservedRunningTime="2026-09-12 01:54:30.503476378 +0000 UTC m=+573.295470512" Sep 12 01:54:30.517686 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:30.517659 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-tl8np"] Sep 12 01:54:30.522907 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:30.522877 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-tl8np"] Sep 12 01:54:31.725838 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:31.725804 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2a37575-b9fb-41ad-b080-d0d9b36f5866" path="/var/lib/kubelet/pods/c2a37575-b9fb-41ad-b080-d0d9b36f5866/volumes" Sep 12 01:54:35.559874 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:35.559781 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:54:35.681256 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:35.681218 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:54:35.772473 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:35.772416 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:54:35.844329 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:35.844245 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:54:38.985977 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:38.985930 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:54:39.077397 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:54:39.077354 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:55:10.476995 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:55:10.476962 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:55:11.619226 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:55:11.619184 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:55:11.951696 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:55:11.951655 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:55:12.036181 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:55:12.036139 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:55:12.098288 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:55:12.098246 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:55:12.317159 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:55:12.317074 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:55:12.510268 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:55:12.510240 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:55:12.625255 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:55:12.625226 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:55:12.883015 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:55:12.882928 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 01:55:26.155465 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:55:26.155340 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:55:26.256239 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:55:26.256195 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:55:29.666853 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:55:29.666818 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:55:30.221904 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:55:30.221868 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 01:55:36.878295 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:55:36.878235 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:55:48.925811 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:55:48.925778 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:55:49.530235 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:55:49.530194 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 01:55:58.820343 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:55:58.820294 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:56:07.434267 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:07.434229 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:56:07.496298 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:07.496262 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:56:15.027910 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:15.027872 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7bgtp"] Sep 12 01:56:15.028408 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:15.028308 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2a37575-b9fb-41ad-b080-d0d9b36f5866" containerName="authorino" Sep 12 01:56:15.028408 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:15.028322 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2a37575-b9fb-41ad-b080-d0d9b36f5866" containerName="authorino" Sep 12 01:56:15.028408 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:15.028391 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2a37575-b9fb-41ad-b080-d0d9b36f5866" containerName="authorino" Sep 12 01:56:15.031579 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:15.031564 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7bgtp" Sep 12 01:56:15.033964 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:15.033942 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 12 01:56:15.034827 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:15.034800 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 12 01:56:15.034827 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:15.034810 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-dzfh7\"" Sep 12 01:56:15.034827 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:15.034817 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Sep 12 01:56:15.039341 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:15.039207 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7bgtp"] Sep 12 01:56:15.096710 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:15.092977 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5fc4975-3ef2-4488-99ed-1e4f50230ab0-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-7bgtp\" (UID: \"b5fc4975-3ef2-4488-99ed-1e4f50230ab0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7bgtp" Sep 12 01:56:15.096710 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:15.093047 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5fc4975-3ef2-4488-99ed-1e4f50230ab0-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-7bgtp\" (UID: \"b5fc4975-3ef2-4488-99ed-1e4f50230ab0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7bgtp" Sep 12 01:56:15.096710 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:15.093082 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5fc4975-3ef2-4488-99ed-1e4f50230ab0-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-7bgtp\" (UID: \"b5fc4975-3ef2-4488-99ed-1e4f50230ab0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7bgtp" Sep 12 01:56:15.096710 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:15.093148 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5fc4975-3ef2-4488-99ed-1e4f50230ab0-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-7bgtp\" (UID: \"b5fc4975-3ef2-4488-99ed-1e4f50230ab0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7bgtp" Sep 12 01:56:15.096710 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:15.093182 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlhxr\" (UniqueName: \"kubernetes.io/projected/b5fc4975-3ef2-4488-99ed-1e4f50230ab0-kube-api-access-dlhxr\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-7bgtp\" (UID: \"b5fc4975-3ef2-4488-99ed-1e4f50230ab0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7bgtp" Sep 12 01:56:15.096710 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:15.093243 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5fc4975-3ef2-4488-99ed-1e4f50230ab0-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-7bgtp\" (UID: \"b5fc4975-3ef2-4488-99ed-1e4f50230ab0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7bgtp" Sep 12 01:56:15.125086 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:15.125050 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:56:15.194367 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:15.194323 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5fc4975-3ef2-4488-99ed-1e4f50230ab0-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-7bgtp\" (UID: \"b5fc4975-3ef2-4488-99ed-1e4f50230ab0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7bgtp" Sep 12 01:56:15.194574 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:15.194383 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5fc4975-3ef2-4488-99ed-1e4f50230ab0-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-7bgtp\" (UID: \"b5fc4975-3ef2-4488-99ed-1e4f50230ab0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7bgtp" Sep 12 01:56:15.194574 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:15.194408 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5fc4975-3ef2-4488-99ed-1e4f50230ab0-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-7bgtp\" (UID: \"b5fc4975-3ef2-4488-99ed-1e4f50230ab0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7bgtp" Sep 12 01:56:15.194574 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:15.194489 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5fc4975-3ef2-4488-99ed-1e4f50230ab0-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-7bgtp\" (UID: \"b5fc4975-3ef2-4488-99ed-1e4f50230ab0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7bgtp" Sep 12 01:56:15.194574 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:15.194521 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlhxr\" (UniqueName: \"kubernetes.io/projected/b5fc4975-3ef2-4488-99ed-1e4f50230ab0-kube-api-access-dlhxr\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-7bgtp\" (UID: \"b5fc4975-3ef2-4488-99ed-1e4f50230ab0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7bgtp" Sep 12 01:56:15.194793 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:15.194673 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5fc4975-3ef2-4488-99ed-1e4f50230ab0-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-7bgtp\" (UID: \"b5fc4975-3ef2-4488-99ed-1e4f50230ab0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7bgtp" Sep 12 01:56:15.194849 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:15.194834 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5fc4975-3ef2-4488-99ed-1e4f50230ab0-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-7bgtp\" (UID: \"b5fc4975-3ef2-4488-99ed-1e4f50230ab0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7bgtp" Sep 12 01:56:15.194957 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:15.194937 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5fc4975-3ef2-4488-99ed-1e4f50230ab0-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-7bgtp\" (UID: \"b5fc4975-3ef2-4488-99ed-1e4f50230ab0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7bgtp" Sep 12 01:56:15.195018 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:15.194995 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5fc4975-3ef2-4488-99ed-1e4f50230ab0-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-7bgtp\" (UID: \"b5fc4975-3ef2-4488-99ed-1e4f50230ab0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7bgtp" Sep 12 01:56:15.196985 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:15.196963 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5fc4975-3ef2-4488-99ed-1e4f50230ab0-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-7bgtp\" (UID: \"b5fc4975-3ef2-4488-99ed-1e4f50230ab0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7bgtp" Sep 12 01:56:15.197142 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:15.197125 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5fc4975-3ef2-4488-99ed-1e4f50230ab0-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-7bgtp\" (UID: \"b5fc4975-3ef2-4488-99ed-1e4f50230ab0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7bgtp" Sep 12 01:56:15.203953 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:15.203917 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlhxr\" (UniqueName: \"kubernetes.io/projected/b5fc4975-3ef2-4488-99ed-1e4f50230ab0-kube-api-access-dlhxr\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-7bgtp\" (UID: \"b5fc4975-3ef2-4488-99ed-1e4f50230ab0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7bgtp" Sep 12 01:56:15.342732 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:15.342646 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7bgtp" Sep 12 01:56:15.615428 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:15.615149 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7bgtp"] Sep 12 01:56:15.617777 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:56:15.617751 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5fc4975_3ef2_4488_99ed_1e4f50230ab0.slice/crio-fc174c039969594ea4affcce35fa0dfb342af92cc949826557d5e58b32d3ce3b WatchSource:0}: Error finding container fc174c039969594ea4affcce35fa0dfb342af92cc949826557d5e58b32d3ce3b: Status 404 returned error can't find the container with id fc174c039969594ea4affcce35fa0dfb342af92cc949826557d5e58b32d3ce3b Sep 12 01:56:15.619499 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:15.619477 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 12 01:56:15.925466 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:15.925415 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 01:56:16.001625 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:16.001584 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7bgtp" event={"ID":"b5fc4975-3ef2-4488-99ed-1e4f50230ab0","Type":"ContainerStarted","Data":"fc174c039969594ea4affcce35fa0dfb342af92cc949826557d5e58b32d3ce3b"} Sep 12 01:56:17.007755 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:17.007665 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7bgtp" event={"ID":"b5fc4975-3ef2-4488-99ed-1e4f50230ab0","Type":"ContainerStarted","Data":"f718be93448617c6199d5c7155e299b9c58d53d4f770a7571fff8e246eafc821"} Sep 12 01:56:17.030475 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:17.028047 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7bgtp" podStartSLOduration=0.936946907 podStartE2EDuration="2.028022496s" totalImagesPullingTime="1.091075589s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:56:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:56:17.027103173 +0000 UTC m=+679.819096978" watchObservedRunningTime="2026-09-12 01:56:17.028022496 +0000 UTC m=+679.820016292" Sep 12 01:56:22.619898 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:22.619859 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:56:23.915984 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:23.915949 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 01:56:25.343393 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:25.343342 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7bgtp" Sep 12 01:56:25.343826 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:25.343415 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7bgtp" Sep 12 01:56:25.356014 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:25.355986 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7bgtp" Sep 12 01:56:26.064526 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:26.064493 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7bgtp" Sep 12 01:56:29.757323 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:29.757284 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:56:31.335532 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:31.335499 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 01:56:36.020790 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:36.020755 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-vrrgt"] Sep 12 01:56:36.024677 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:36.024650 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vrrgt" Sep 12 01:56:36.027149 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:36.027121 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Sep 12 01:56:36.037253 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:36.037227 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-vrrgt"] Sep 12 01:56:36.049787 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:36.049752 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:56:36.090685 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:36.090642 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0297867c-0e30-4053-a687-b38a6d25ed52-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vrrgt\" (UID: \"0297867c-0e30-4053-a687-b38a6d25ed52\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vrrgt" Sep 12 01:56:36.090685 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:36.090686 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0297867c-0e30-4053-a687-b38a6d25ed52-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vrrgt\" (UID: \"0297867c-0e30-4053-a687-b38a6d25ed52\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vrrgt" Sep 12 01:56:36.090935 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:36.090709 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq58f\" (UniqueName: \"kubernetes.io/projected/0297867c-0e30-4053-a687-b38a6d25ed52-kube-api-access-fq58f\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vrrgt\" (UID: \"0297867c-0e30-4053-a687-b38a6d25ed52\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vrrgt" Sep 12 01:56:36.090935 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:36.090852 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0297867c-0e30-4053-a687-b38a6d25ed52-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vrrgt\" (UID: \"0297867c-0e30-4053-a687-b38a6d25ed52\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vrrgt" Sep 12 01:56:36.090935 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:36.090879 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0297867c-0e30-4053-a687-b38a6d25ed52-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vrrgt\" (UID: \"0297867c-0e30-4053-a687-b38a6d25ed52\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vrrgt" Sep 12 01:56:36.090935 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:36.090896 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0297867c-0e30-4053-a687-b38a6d25ed52-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vrrgt\" (UID: \"0297867c-0e30-4053-a687-b38a6d25ed52\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vrrgt" Sep 12 01:56:36.192201 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:36.192156 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0297867c-0e30-4053-a687-b38a6d25ed52-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vrrgt\" (UID: \"0297867c-0e30-4053-a687-b38a6d25ed52\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vrrgt" Sep 12 01:56:36.192201 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:36.192201 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0297867c-0e30-4053-a687-b38a6d25ed52-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vrrgt\" (UID: \"0297867c-0e30-4053-a687-b38a6d25ed52\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vrrgt" Sep 12 01:56:36.192495 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:36.192219 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0297867c-0e30-4053-a687-b38a6d25ed52-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vrrgt\" (UID: \"0297867c-0e30-4053-a687-b38a6d25ed52\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vrrgt" Sep 12 01:56:36.192495 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:36.192266 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0297867c-0e30-4053-a687-b38a6d25ed52-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vrrgt\" (UID: \"0297867c-0e30-4053-a687-b38a6d25ed52\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vrrgt" Sep 12 01:56:36.192495 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:36.192290 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0297867c-0e30-4053-a687-b38a6d25ed52-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vrrgt\" (UID: \"0297867c-0e30-4053-a687-b38a6d25ed52\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vrrgt" Sep 12 01:56:36.192495 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:36.192308 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq58f\" (UniqueName: \"kubernetes.io/projected/0297867c-0e30-4053-a687-b38a6d25ed52-kube-api-access-fq58f\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vrrgt\" (UID: \"0297867c-0e30-4053-a687-b38a6d25ed52\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vrrgt" Sep 12 01:56:36.192713 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:36.192611 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0297867c-0e30-4053-a687-b38a6d25ed52-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vrrgt\" (UID: \"0297867c-0e30-4053-a687-b38a6d25ed52\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vrrgt" Sep 12 01:56:36.192713 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:36.192684 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0297867c-0e30-4053-a687-b38a6d25ed52-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vrrgt\" (UID: \"0297867c-0e30-4053-a687-b38a6d25ed52\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vrrgt" Sep 12 01:56:36.192791 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:36.192730 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0297867c-0e30-4053-a687-b38a6d25ed52-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vrrgt\" (UID: \"0297867c-0e30-4053-a687-b38a6d25ed52\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vrrgt" Sep 12 01:56:36.194745 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:36.194723 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0297867c-0e30-4053-a687-b38a6d25ed52-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vrrgt\" (UID: \"0297867c-0e30-4053-a687-b38a6d25ed52\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vrrgt" Sep 12 01:56:36.195016 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:36.194998 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0297867c-0e30-4053-a687-b38a6d25ed52-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vrrgt\" (UID: \"0297867c-0e30-4053-a687-b38a6d25ed52\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vrrgt" Sep 12 01:56:36.200513 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:36.200491 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq58f\" (UniqueName: \"kubernetes.io/projected/0297867c-0e30-4053-a687-b38a6d25ed52-kube-api-access-fq58f\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vrrgt\" (UID: \"0297867c-0e30-4053-a687-b38a6d25ed52\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vrrgt" Sep 12 01:56:36.335331 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:36.335241 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vrrgt" Sep 12 01:56:36.609834 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:36.609781 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-vrrgt"] Sep 12 01:56:36.612131 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:56:36.612102 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0297867c_0e30_4053_a687_b38a6d25ed52.slice/crio-3999a66375afb997a4adcacc37cb8fc24dfa9fbefef4bc9e05616ad3aa2138a9 WatchSource:0}: Error finding container 3999a66375afb997a4adcacc37cb8fc24dfa9fbefef4bc9e05616ad3aa2138a9: Status 404 returned error can't find the container with id 3999a66375afb997a4adcacc37cb8fc24dfa9fbefef4bc9e05616ad3aa2138a9 Sep 12 01:56:37.104468 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:37.104428 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vrrgt" event={"ID":"0297867c-0e30-4053-a687-b38a6d25ed52","Type":"ContainerStarted","Data":"573110e1254ae183c855d666b8fb0a96413e22433630de0dbdca762cdb75a3b9"} Sep 12 01:56:37.104883 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:37.104501 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vrrgt" event={"ID":"0297867c-0e30-4053-a687-b38a6d25ed52","Type":"ContainerStarted","Data":"3999a66375afb997a4adcacc37cb8fc24dfa9fbefef4bc9e05616ad3aa2138a9"} Sep 12 01:56:37.125755 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:37.125706 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vrrgt" podStartSLOduration=0.928715011 podStartE2EDuration="1.12568397s" totalImagesPullingTime="196.968959ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:56:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:56:37.123668426 +0000 UTC m=+699.915662222" watchObservedRunningTime="2026-09-12 01:56:37.12568397 +0000 UTC m=+699.917677972" Sep 12 01:56:38.029181 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:38.029133 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 01:56:46.335552 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:46.335515 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vrrgt" Sep 12 01:56:46.336103 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:46.335571 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vrrgt" Sep 12 01:56:46.348084 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:46.348060 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vrrgt" Sep 12 01:56:47.169142 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:47.169115 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vrrgt" Sep 12 01:56:57.017289 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:57.017211 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-29qsh"] Sep 12 01:56:57.021527 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:57.021511 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-29qsh" Sep 12 01:56:57.024017 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:57.023994 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Sep 12 01:56:57.029985 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:57.029963 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-29qsh"] Sep 12 01:56:57.137847 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:57.137815 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:56:57.184905 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:57.184876 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3311a5d0-7320-4533-b6dc-56687a3a1028-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-29qsh\" (UID: \"3311a5d0-7320-4533-b6dc-56687a3a1028\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-29qsh" Sep 12 01:56:57.184905 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:57.184905 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3311a5d0-7320-4533-b6dc-56687a3a1028-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-29qsh\" (UID: \"3311a5d0-7320-4533-b6dc-56687a3a1028\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-29qsh" Sep 12 01:56:57.185100 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:57.184926 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r847f\" (UniqueName: \"kubernetes.io/projected/3311a5d0-7320-4533-b6dc-56687a3a1028-kube-api-access-r847f\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-29qsh\" (UID: \"3311a5d0-7320-4533-b6dc-56687a3a1028\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-29qsh" Sep 12 01:56:57.185100 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:57.185029 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3311a5d0-7320-4533-b6dc-56687a3a1028-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-29qsh\" (UID: \"3311a5d0-7320-4533-b6dc-56687a3a1028\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-29qsh" Sep 12 01:56:57.185100 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:57.185058 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3311a5d0-7320-4533-b6dc-56687a3a1028-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-29qsh\" (UID: \"3311a5d0-7320-4533-b6dc-56687a3a1028\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-29qsh" Sep 12 01:56:57.185100 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:57.185080 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3311a5d0-7320-4533-b6dc-56687a3a1028-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-29qsh\" (UID: \"3311a5d0-7320-4533-b6dc-56687a3a1028\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-29qsh" Sep 12 01:56:57.286485 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:57.286391 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3311a5d0-7320-4533-b6dc-56687a3a1028-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-29qsh\" (UID: \"3311a5d0-7320-4533-b6dc-56687a3a1028\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-29qsh" Sep 12 01:56:57.286485 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:57.286423 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3311a5d0-7320-4533-b6dc-56687a3a1028-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-29qsh\" (UID: \"3311a5d0-7320-4533-b6dc-56687a3a1028\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-29qsh" Sep 12 01:56:57.286485 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:57.286441 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r847f\" (UniqueName: \"kubernetes.io/projected/3311a5d0-7320-4533-b6dc-56687a3a1028-kube-api-access-r847f\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-29qsh\" (UID: \"3311a5d0-7320-4533-b6dc-56687a3a1028\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-29qsh" Sep 12 01:56:57.286758 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:57.286518 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3311a5d0-7320-4533-b6dc-56687a3a1028-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-29qsh\" (UID: \"3311a5d0-7320-4533-b6dc-56687a3a1028\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-29qsh" Sep 12 01:56:57.286758 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:57.286536 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3311a5d0-7320-4533-b6dc-56687a3a1028-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-29qsh\" (UID: \"3311a5d0-7320-4533-b6dc-56687a3a1028\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-29qsh" Sep 12 01:56:57.286758 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:57.286555 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3311a5d0-7320-4533-b6dc-56687a3a1028-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-29qsh\" (UID: \"3311a5d0-7320-4533-b6dc-56687a3a1028\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-29qsh" Sep 12 01:56:57.286922 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:57.286886 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3311a5d0-7320-4533-b6dc-56687a3a1028-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-29qsh\" (UID: \"3311a5d0-7320-4533-b6dc-56687a3a1028\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-29qsh" Sep 12 01:56:57.286961 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:57.286887 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3311a5d0-7320-4533-b6dc-56687a3a1028-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-29qsh\" (UID: \"3311a5d0-7320-4533-b6dc-56687a3a1028\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-29qsh" Sep 12 01:56:57.286961 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:57.286939 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3311a5d0-7320-4533-b6dc-56687a3a1028-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-29qsh\" (UID: \"3311a5d0-7320-4533-b6dc-56687a3a1028\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-29qsh" Sep 12 01:56:57.288801 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:57.288784 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3311a5d0-7320-4533-b6dc-56687a3a1028-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-29qsh\" (UID: \"3311a5d0-7320-4533-b6dc-56687a3a1028\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-29qsh" Sep 12 01:56:57.292513 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:57.289471 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3311a5d0-7320-4533-b6dc-56687a3a1028-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-29qsh\" (UID: \"3311a5d0-7320-4533-b6dc-56687a3a1028\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-29qsh" Sep 12 01:56:57.295938 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:57.295905 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r847f\" (UniqueName: \"kubernetes.io/projected/3311a5d0-7320-4533-b6dc-56687a3a1028-kube-api-access-r847f\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-29qsh\" (UID: \"3311a5d0-7320-4533-b6dc-56687a3a1028\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-29qsh" Sep 12 01:56:57.331368 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:57.331341 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-29qsh" Sep 12 01:56:57.606327 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:57.606215 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-29qsh"] Sep 12 01:56:57.608921 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:56:57.608894 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3311a5d0_7320_4533_b6dc_56687a3a1028.slice/crio-4b3582058d7866c48f6ab230ba581a73e72d55a4fde8be0adbfd63b2fff97d0c WatchSource:0}: Error finding container 4b3582058d7866c48f6ab230ba581a73e72d55a4fde8be0adbfd63b2fff97d0c: Status 404 returned error can't find the container with id 4b3582058d7866c48f6ab230ba581a73e72d55a4fde8be0adbfd63b2fff97d0c Sep 12 01:56:58.210103 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:58.210062 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-29qsh" event={"ID":"3311a5d0-7320-4533-b6dc-56687a3a1028","Type":"ContainerStarted","Data":"8fe2ccc16d10d5586e0742b63bf4a433d92ba97b198e1dfa4d85f7f80581fed8"} Sep 12 01:56:58.210103 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:58.210101 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-29qsh" event={"ID":"3311a5d0-7320-4533-b6dc-56687a3a1028","Type":"ContainerStarted","Data":"4b3582058d7866c48f6ab230ba581a73e72d55a4fde8be0adbfd63b2fff97d0c"} Sep 12 01:56:58.228440 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:58.228385 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-29qsh" podStartSLOduration=2.043928788 podStartE2EDuration="2.228368473s" totalImagesPullingTime="184.439685ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:56:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:56:58.226575649 +0000 UTC m=+721.018569450" watchObservedRunningTime="2026-09-12 01:56:58.228368473 +0000 UTC m=+721.020362268" Sep 12 01:56:59.536532 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:56:59.536496 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 01:57:07.332143 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:57:07.332108 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-29qsh" Sep 12 01:57:07.332659 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:57:07.332169 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-29qsh" Sep 12 01:57:07.344641 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:57:07.344615 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-29qsh" Sep 12 01:57:08.275904 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:57:08.275875 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-29qsh" Sep 12 01:57:25.615369 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:57:25.615333 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:57:25.674343 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:57:25.674307 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:57:38.990179 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:57:38.990140 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:57:43.349036 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:57:43.349001 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56bb445f59-4mxgl"] Sep 12 01:57:43.352816 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:57:43.352786 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56bb445f59-4mxgl" Sep 12 01:57:43.359185 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:57:43.359157 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56bb445f59-4mxgl"] Sep 12 01:57:43.422993 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:57:43.422962 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1393128d-7b35-4819-83ee-048251caa3d0-tls-cert\") pod \"authorino-56bb445f59-4mxgl\" (UID: \"1393128d-7b35-4819-83ee-048251caa3d0\") " pod="kuadrant-system/authorino-56bb445f59-4mxgl" Sep 12 01:57:43.423175 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:57:43.423126 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xfvl\" (UniqueName: \"kubernetes.io/projected/1393128d-7b35-4819-83ee-048251caa3d0-kube-api-access-6xfvl\") pod \"authorino-56bb445f59-4mxgl\" (UID: \"1393128d-7b35-4819-83ee-048251caa3d0\") " pod="kuadrant-system/authorino-56bb445f59-4mxgl" Sep 12 01:57:43.524004 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:57:43.523961 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xfvl\" (UniqueName: \"kubernetes.io/projected/1393128d-7b35-4819-83ee-048251caa3d0-kube-api-access-6xfvl\") pod \"authorino-56bb445f59-4mxgl\" (UID: \"1393128d-7b35-4819-83ee-048251caa3d0\") " pod="kuadrant-system/authorino-56bb445f59-4mxgl" Sep 12 01:57:43.524004 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:57:43.524012 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1393128d-7b35-4819-83ee-048251caa3d0-tls-cert\") pod \"authorino-56bb445f59-4mxgl\" (UID: \"1393128d-7b35-4819-83ee-048251caa3d0\") " pod="kuadrant-system/authorino-56bb445f59-4mxgl" Sep 12 01:57:43.526624 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:57:43.526605 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1393128d-7b35-4819-83ee-048251caa3d0-tls-cert\") pod \"authorino-56bb445f59-4mxgl\" (UID: \"1393128d-7b35-4819-83ee-048251caa3d0\") " pod="kuadrant-system/authorino-56bb445f59-4mxgl" Sep 12 01:57:43.532212 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:57:43.532190 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xfvl\" (UniqueName: \"kubernetes.io/projected/1393128d-7b35-4819-83ee-048251caa3d0-kube-api-access-6xfvl\") pod \"authorino-56bb445f59-4mxgl\" (UID: \"1393128d-7b35-4819-83ee-048251caa3d0\") " pod="kuadrant-system/authorino-56bb445f59-4mxgl" Sep 12 01:57:43.664299 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:57:43.664272 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56bb445f59-4mxgl" Sep 12 01:57:43.932581 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:57:43.932553 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56bb445f59-4mxgl"] Sep 12 01:57:43.934207 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:57:43.934178 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1393128d_7b35_4819_83ee_048251caa3d0.slice/crio-3c230aba95a4384fb0ca4aab7bb05638d0093c62a98d4626d2ef319f691f1bb5 WatchSource:0}: Error finding container 3c230aba95a4384fb0ca4aab7bb05638d0093c62a98d4626d2ef319f691f1bb5: Status 404 returned error can't find the container with id 3c230aba95a4384fb0ca4aab7bb05638d0093c62a98d4626d2ef319f691f1bb5 Sep 12 01:57:44.435743 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:57:44.435713 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56bb445f59-4mxgl" event={"ID":"1393128d-7b35-4819-83ee-048251caa3d0","Type":"ContainerStarted","Data":"3c230aba95a4384fb0ca4aab7bb05638d0093c62a98d4626d2ef319f691f1bb5"} Sep 12 01:57:45.445053 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:57:45.445010 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56bb445f59-4mxgl" event={"ID":"1393128d-7b35-4819-83ee-048251caa3d0","Type":"ContainerStarted","Data":"e6235193aa1ca479d1f3a08a9549c8113b86cc962f34c7301db46977bbdd8bf2"} Sep 12 01:57:45.462857 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:57:45.462796 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-56bb445f59-4mxgl" podStartSLOduration=1.714543392 podStartE2EDuration="2.462773963s" totalImagesPullingTime="748.230571ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:57:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:57:45.461872795 +0000 UTC m=+768.253866591" watchObservedRunningTime="2026-09-12 01:57:45.462773963 +0000 UTC m=+768.254767759" Sep 12 01:57:45.489418 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:57:45.489385 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6fc68b8b94-8gl2b"] Sep 12 01:57:45.489626 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:57:45.489590 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-6fc68b8b94-8gl2b" podUID="fd8bf9f4-8be6-4918-8a22-511605c38365" containerName="authorino" containerID="cri-o://f23296ab11ca8aa833b61395a8817f5326767d0fe5ebde8bc9bba2fff73acd00" gracePeriod=30 Sep 12 01:57:45.756992 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:57:45.756968 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6fc68b8b94-8gl2b" Sep 12 01:57:45.850362 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:57:45.850330 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fd8bf9f4-8be6-4918-8a22-511605c38365-tls-cert\") pod \"fd8bf9f4-8be6-4918-8a22-511605c38365\" (UID: \"fd8bf9f4-8be6-4918-8a22-511605c38365\") " Sep 12 01:57:45.850575 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:57:45.850447 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-68k7r\" (UniqueName: \"kubernetes.io/projected/fd8bf9f4-8be6-4918-8a22-511605c38365-kube-api-access-68k7r\") pod \"fd8bf9f4-8be6-4918-8a22-511605c38365\" (UID: \"fd8bf9f4-8be6-4918-8a22-511605c38365\") " Sep 12 01:57:45.852715 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:57:45.852685 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd8bf9f4-8be6-4918-8a22-511605c38365-kube-api-access-68k7r" (OuterVolumeSpecName: "kube-api-access-68k7r") pod "fd8bf9f4-8be6-4918-8a22-511605c38365" (UID: "fd8bf9f4-8be6-4918-8a22-511605c38365"). InnerVolumeSpecName "kube-api-access-68k7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 12 01:57:45.864758 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:57:45.864727 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd8bf9f4-8be6-4918-8a22-511605c38365-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "fd8bf9f4-8be6-4918-8a22-511605c38365" (UID: "fd8bf9f4-8be6-4918-8a22-511605c38365"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 12 01:57:45.951506 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:57:45.951445 2579 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fd8bf9f4-8be6-4918-8a22-511605c38365-tls-cert\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 01:57:45.951506 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:57:45.951496 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68k7r\" (UniqueName: \"kubernetes.io/projected/fd8bf9f4-8be6-4918-8a22-511605c38365-kube-api-access-68k7r\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 01:57:46.034831 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:57:46.034728 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:57:46.129668 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:57:46.129631 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:57:46.451586 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:57:46.451555 2579 generic.go:356] "Generic (PLEG): container finished" podID="fd8bf9f4-8be6-4918-8a22-511605c38365" containerID="f23296ab11ca8aa833b61395a8817f5326767d0fe5ebde8bc9bba2fff73acd00" exitCode=0 Sep 12 01:57:46.451977 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:57:46.451609 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6fc68b8b94-8gl2b" Sep 12 01:57:46.451977 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:57:46.451648 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6fc68b8b94-8gl2b" event={"ID":"fd8bf9f4-8be6-4918-8a22-511605c38365","Type":"ContainerDied","Data":"f23296ab11ca8aa833b61395a8817f5326767d0fe5ebde8bc9bba2fff73acd00"} Sep 12 01:57:46.451977 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:57:46.451696 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6fc68b8b94-8gl2b" event={"ID":"fd8bf9f4-8be6-4918-8a22-511605c38365","Type":"ContainerDied","Data":"7c596d6f1285b2d20a3a896e0ade99db98244c2ba59fe8a7f3efb6e613d937c2"} Sep 12 01:57:46.451977 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:57:46.451722 2579 scope.go:117] "RemoveContainer" containerID="f23296ab11ca8aa833b61395a8817f5326767d0fe5ebde8bc9bba2fff73acd00" Sep 12 01:57:46.461630 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:57:46.461616 2579 scope.go:117] "RemoveContainer" containerID="f23296ab11ca8aa833b61395a8817f5326767d0fe5ebde8bc9bba2fff73acd00" Sep 12 01:57:46.461920 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:57:46.461903 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f23296ab11ca8aa833b61395a8817f5326767d0fe5ebde8bc9bba2fff73acd00\": container with ID starting with f23296ab11ca8aa833b61395a8817f5326767d0fe5ebde8bc9bba2fff73acd00 not found: ID does not exist" containerID="f23296ab11ca8aa833b61395a8817f5326767d0fe5ebde8bc9bba2fff73acd00" Sep 12 01:57:46.461979 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:57:46.461929 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f23296ab11ca8aa833b61395a8817f5326767d0fe5ebde8bc9bba2fff73acd00"} err="failed to get container status \"f23296ab11ca8aa833b61395a8817f5326767d0fe5ebde8bc9bba2fff73acd00\": rpc error: code = NotFound desc = could not find container \"f23296ab11ca8aa833b61395a8817f5326767d0fe5ebde8bc9bba2fff73acd00\": container with ID starting with f23296ab11ca8aa833b61395a8817f5326767d0fe5ebde8bc9bba2fff73acd00 not found: ID does not exist" Sep 12 01:57:46.477638 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:57:46.477610 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6fc68b8b94-8gl2b"] Sep 12 01:57:46.484825 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:57:46.484802 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6fc68b8b94-8gl2b"] Sep 12 01:57:47.222767 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:57:47.222730 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:57:47.320262 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:57:47.320228 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:57:47.725895 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:57:47.725867 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd8bf9f4-8be6-4918-8a22-511605c38365" path="/var/lib/kubelet/pods/fd8bf9f4-8be6-4918-8a22-511605c38365/volumes" Sep 12 01:58:03.161291 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:03.161241 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:03.261275 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:03.261238 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:03.363518 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:03.363478 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:03.538481 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:03.538362 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:03.620052 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:03.620018 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:03.715740 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:03.715692 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:03.815733 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:03.815636 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:04.040113 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:04.040077 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:04.108259 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:04.108172 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:04.657336 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:04.657300 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:04.816805 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:04.816768 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:05.160085 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:05.160051 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:06.872852 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:06.872816 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:06.965973 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:06.965938 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:07.075721 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:07.075682 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:07.172572 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:07.172482 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:07.230147 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:07.230113 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 01:58:08.399240 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:08.399207 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:08.500170 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:08.500125 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:08.601358 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:08.601320 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:08.996121 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:08.996086 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:09.075854 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:09.071433 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:09.075854 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:09.072487 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f660a6-7c79ccd4b6-l6sfk"] Sep 12 01:58:09.075854 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:09.073397 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd8bf9f4-8be6-4918-8a22-511605c38365" containerName="authorino" Sep 12 01:58:09.075854 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:09.073414 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd8bf9f4-8be6-4918-8a22-511605c38365" containerName="authorino" Sep 12 01:58:09.075854 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:09.073624 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd8bf9f4-8be6-4918-8a22-511605c38365" containerName="authorino" Sep 12 01:58:09.078961 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:09.078939 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f660a6-7c79ccd4b6-l6sfk" Sep 12 01:58:09.081341 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:09.081322 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-f660a6-dockercfg-4llpm\"" Sep 12 01:58:09.081470 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:09.081350 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-f660a6\"" Sep 12 01:58:09.085432 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:09.085409 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f660a6-7c79ccd4b6-l6sfk"] Sep 12 01:58:09.178343 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:09.178294 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bmqt\" (UniqueName: \"kubernetes.io/projected/c273cea1-7278-4513-8432-ccba4c25c999-kube-api-access-2bmqt\") pod \"payload-pre-processing-e2e-worker-w0-f660a6-7c79ccd4b6-l6sfk\" (UID: \"c273cea1-7278-4513-8432-ccba4c25c999\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f660a6-7c79ccd4b6-l6sfk" Sep 12 01:58:09.178557 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:09.178355 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c273cea1-7278-4513-8432-ccba4c25c999-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-f660a6-7c79ccd4b6-l6sfk\" (UID: \"c273cea1-7278-4513-8432-ccba4c25c999\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f660a6-7c79ccd4b6-l6sfk" Sep 12 01:58:09.279727 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:09.279619 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bmqt\" (UniqueName: \"kubernetes.io/projected/c273cea1-7278-4513-8432-ccba4c25c999-kube-api-access-2bmqt\") pod \"payload-pre-processing-e2e-worker-w0-f660a6-7c79ccd4b6-l6sfk\" (UID: \"c273cea1-7278-4513-8432-ccba4c25c999\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f660a6-7c79ccd4b6-l6sfk" Sep 12 01:58:09.279727 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:09.279676 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c273cea1-7278-4513-8432-ccba4c25c999-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-f660a6-7c79ccd4b6-l6sfk\" (UID: \"c273cea1-7278-4513-8432-ccba4c25c999\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f660a6-7c79ccd4b6-l6sfk" Sep 12 01:58:09.280468 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:09.280429 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c273cea1-7278-4513-8432-ccba4c25c999-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-f660a6-7c79ccd4b6-l6sfk\" (UID: \"c273cea1-7278-4513-8432-ccba4c25c999\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f660a6-7c79ccd4b6-l6sfk" Sep 12 01:58:09.287962 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:09.287934 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bmqt\" (UniqueName: \"kubernetes.io/projected/c273cea1-7278-4513-8432-ccba4c25c999-kube-api-access-2bmqt\") pod \"payload-pre-processing-e2e-worker-w0-f660a6-7c79ccd4b6-l6sfk\" (UID: \"c273cea1-7278-4513-8432-ccba4c25c999\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f660a6-7c79ccd4b6-l6sfk" Sep 12 01:58:09.342803 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:09.342767 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3fc028-668b6db875-svkb5"] Sep 12 01:58:09.347376 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:09.347349 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3fc028-668b6db875-svkb5" Sep 12 01:58:09.349804 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:09.349769 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-3fc028-serving-cert\"" Sep 12 01:58:09.350008 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:09.349986 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-565nn\"" Sep 12 01:58:09.350109 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:09.350048 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 12 01:58:09.350342 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:09.350323 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 12 01:58:09.350528 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:09.350506 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 12 01:58:09.357374 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:09.357352 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3fc028-668b6db875-svkb5"] Sep 12 01:58:09.378708 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:09.378678 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:09.380809 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:09.380790 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/268dd63e-c941-4687-ba08-f4e342b2585f-maas-api-tls\") pod \"maas-api-e2e-inf-a-3fc028-668b6db875-svkb5\" (UID: \"268dd63e-c941-4687-ba08-f4e342b2585f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3fc028-668b6db875-svkb5" Sep 12 01:58:09.380921 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:09.380908 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txprt\" (UniqueName: \"kubernetes.io/projected/268dd63e-c941-4687-ba08-f4e342b2585f-kube-api-access-txprt\") pod \"maas-api-e2e-inf-a-3fc028-668b6db875-svkb5\" (UID: \"268dd63e-c941-4687-ba08-f4e342b2585f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3fc028-668b6db875-svkb5" Sep 12 01:58:09.380965 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:09.380938 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/268dd63e-c941-4687-ba08-f4e342b2585f-metrics-certs\") pod \"maas-api-e2e-inf-a-3fc028-668b6db875-svkb5\" (UID: \"268dd63e-c941-4687-ba08-f4e342b2585f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3fc028-668b6db875-svkb5" Sep 12 01:58:09.381020 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:09.380964 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/268dd63e-c941-4687-ba08-f4e342b2585f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3fc028-668b6db875-svkb5\" (UID: \"268dd63e-c941-4687-ba08-f4e342b2585f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3fc028-668b6db875-svkb5" Sep 12 01:58:09.391714 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:09.391685 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f660a6-7c79ccd4b6-l6sfk" Sep 12 01:58:09.467513 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:09.467439 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:09.481784 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:09.481748 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txprt\" (UniqueName: \"kubernetes.io/projected/268dd63e-c941-4687-ba08-f4e342b2585f-kube-api-access-txprt\") pod \"maas-api-e2e-inf-a-3fc028-668b6db875-svkb5\" (UID: \"268dd63e-c941-4687-ba08-f4e342b2585f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3fc028-668b6db875-svkb5" Sep 12 01:58:09.481784 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:09.481789 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/268dd63e-c941-4687-ba08-f4e342b2585f-metrics-certs\") pod \"maas-api-e2e-inf-a-3fc028-668b6db875-svkb5\" (UID: \"268dd63e-c941-4687-ba08-f4e342b2585f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3fc028-668b6db875-svkb5" Sep 12 01:58:09.482038 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:09.481808 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/268dd63e-c941-4687-ba08-f4e342b2585f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3fc028-668b6db875-svkb5\" (UID: \"268dd63e-c941-4687-ba08-f4e342b2585f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3fc028-668b6db875-svkb5" Sep 12 01:58:09.482038 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:09.481850 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/268dd63e-c941-4687-ba08-f4e342b2585f-maas-api-tls\") pod \"maas-api-e2e-inf-a-3fc028-668b6db875-svkb5\" (UID: \"268dd63e-c941-4687-ba08-f4e342b2585f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3fc028-668b6db875-svkb5" Sep 12 01:58:09.482038 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:58:09.481964 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-3fc028-serving-cert: secret "maas-api-e2e-inf-a-3fc028-serving-cert" not found Sep 12 01:58:09.482038 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:58:09.482031 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/268dd63e-c941-4687-ba08-f4e342b2585f-maas-api-tls podName:268dd63e-c941-4687-ba08-f4e342b2585f nodeName:}" failed. No retries permitted until 2026-09-12 01:58:09.982009965 +0000 UTC m=+792.774003740 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/268dd63e-c941-4687-ba08-f4e342b2585f-maas-api-tls") pod "maas-api-e2e-inf-a-3fc028-668b6db875-svkb5" (UID: "268dd63e-c941-4687-ba08-f4e342b2585f") : secret "maas-api-e2e-inf-a-3fc028-serving-cert" not found Sep 12 01:58:09.482518 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:09.482501 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/268dd63e-c941-4687-ba08-f4e342b2585f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3fc028-668b6db875-svkb5\" (UID: \"268dd63e-c941-4687-ba08-f4e342b2585f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3fc028-668b6db875-svkb5" Sep 12 01:58:09.484404 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:09.484382 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/268dd63e-c941-4687-ba08-f4e342b2585f-metrics-certs\") pod \"maas-api-e2e-inf-a-3fc028-668b6db875-svkb5\" (UID: \"268dd63e-c941-4687-ba08-f4e342b2585f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3fc028-668b6db875-svkb5" Sep 12 01:58:09.504366 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:09.504334 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txprt\" (UniqueName: \"kubernetes.io/projected/268dd63e-c941-4687-ba08-f4e342b2585f-kube-api-access-txprt\") pod \"maas-api-e2e-inf-a-3fc028-668b6db875-svkb5\" (UID: \"268dd63e-c941-4687-ba08-f4e342b2585f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3fc028-668b6db875-svkb5" Sep 12 01:58:09.671638 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:09.671599 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f660a6-7c79ccd4b6-l6sfk"] Sep 12 01:58:09.675650 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:58:09.675623 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc273cea1_7278_4513_8432_ccba4c25c999.slice/crio-5044056010be35fdc691bba838a5745e9f8730535655e23769738d06a8e27aa4 WatchSource:0}: Error finding container 5044056010be35fdc691bba838a5745e9f8730535655e23769738d06a8e27aa4: Status 404 returned error can't find the container with id 5044056010be35fdc691bba838a5745e9f8730535655e23769738d06a8e27aa4 Sep 12 01:58:09.787048 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:09.787003 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:09.987994 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:09.987910 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/268dd63e-c941-4687-ba08-f4e342b2585f-maas-api-tls\") pod \"maas-api-e2e-inf-a-3fc028-668b6db875-svkb5\" (UID: \"268dd63e-c941-4687-ba08-f4e342b2585f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3fc028-668b6db875-svkb5" Sep 12 01:58:09.990738 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:09.990720 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/268dd63e-c941-4687-ba08-f4e342b2585f-maas-api-tls\") pod \"maas-api-e2e-inf-a-3fc028-668b6db875-svkb5\" (UID: \"268dd63e-c941-4687-ba08-f4e342b2585f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3fc028-668b6db875-svkb5" Sep 12 01:58:10.082686 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:10.082650 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:10.179961 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:10.179929 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:10.216584 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:10.216552 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3fc028-67db4f49b5-5jjjf"] Sep 12 01:58:10.222037 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:10.222015 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3fc028-67db4f49b5-5jjjf" Sep 12 01:58:10.224571 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:10.224547 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-3fc028\"" Sep 12 01:58:10.235109 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:10.235080 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3fc028-67db4f49b5-5jjjf"] Sep 12 01:58:10.261051 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:10.260965 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3fc028-668b6db875-svkb5" Sep 12 01:58:10.290797 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:10.290760 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-894qj\" (UniqueName: \"kubernetes.io/projected/6651f43f-9684-460e-b2f6-787e35ebe96d-kube-api-access-894qj\") pod \"payload-processing-e2e-inf-a-3fc028-67db4f49b5-5jjjf\" (UID: \"6651f43f-9684-460e-b2f6-787e35ebe96d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3fc028-67db4f49b5-5jjjf" Sep 12 01:58:10.290963 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:10.290817 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6651f43f-9684-460e-b2f6-787e35ebe96d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3fc028-67db4f49b5-5jjjf\" (UID: \"6651f43f-9684-460e-b2f6-787e35ebe96d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3fc028-67db4f49b5-5jjjf" Sep 12 01:58:10.391676 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:10.391640 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-894qj\" (UniqueName: \"kubernetes.io/projected/6651f43f-9684-460e-b2f6-787e35ebe96d-kube-api-access-894qj\") pod \"payload-processing-e2e-inf-a-3fc028-67db4f49b5-5jjjf\" (UID: \"6651f43f-9684-460e-b2f6-787e35ebe96d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3fc028-67db4f49b5-5jjjf" Sep 12 01:58:10.391867 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:10.391681 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6651f43f-9684-460e-b2f6-787e35ebe96d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3fc028-67db4f49b5-5jjjf\" (UID: \"6651f43f-9684-460e-b2f6-787e35ebe96d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3fc028-67db4f49b5-5jjjf" Sep 12 01:58:10.392442 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:10.392416 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6651f43f-9684-460e-b2f6-787e35ebe96d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3fc028-67db4f49b5-5jjjf\" (UID: \"6651f43f-9684-460e-b2f6-787e35ebe96d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3fc028-67db4f49b5-5jjjf" Sep 12 01:58:10.406106 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:10.406078 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-894qj\" (UniqueName: \"kubernetes.io/projected/6651f43f-9684-460e-b2f6-787e35ebe96d-kube-api-access-894qj\") pod \"payload-processing-e2e-inf-a-3fc028-67db4f49b5-5jjjf\" (UID: \"6651f43f-9684-460e-b2f6-787e35ebe96d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3fc028-67db4f49b5-5jjjf" Sep 12 01:58:10.534464 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:10.534355 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3fc028-67db4f49b5-5jjjf" Sep 12 01:58:10.555311 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:10.554434 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3fc028-668b6db875-svkb5"] Sep 12 01:58:10.556207 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:58:10.556165 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod268dd63e_c941_4687_ba08_f4e342b2585f.slice/crio-5b38489aff8e57d17ad09d4569f2cfae575073c0dda7a233ab67333fa31f76dd WatchSource:0}: Error finding container 5b38489aff8e57d17ad09d4569f2cfae575073c0dda7a233ab67333fa31f76dd: Status 404 returned error can't find the container with id 5b38489aff8e57d17ad09d4569f2cfae575073c0dda7a233ab67333fa31f76dd Sep 12 01:58:10.577548 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:10.577507 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3fc028-668b6db875-svkb5" event={"ID":"268dd63e-c941-4687-ba08-f4e342b2585f","Type":"ContainerStarted","Data":"5b38489aff8e57d17ad09d4569f2cfae575073c0dda7a233ab67333fa31f76dd"} Sep 12 01:58:10.578932 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:10.578891 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f660a6-7c79ccd4b6-l6sfk" event={"ID":"c273cea1-7278-4513-8432-ccba4c25c999","Type":"ContainerStarted","Data":"5044056010be35fdc691bba838a5745e9f8730535655e23769738d06a8e27aa4"} Sep 12 01:58:10.678973 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:10.678921 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:10.782198 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:10.782155 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:10.822556 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:10.822530 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3fc028-67db4f49b5-5jjjf"] Sep 12 01:58:11.075052 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:58:11.074979 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6651f43f_9684_460e_b2f6_787e35ebe96d.slice/crio-c049962f9637a48faf4fbe6f6d69d1521cb3d5a7a6924e7ac6be3cec909e72ad WatchSource:0}: Error finding container c049962f9637a48faf4fbe6f6d69d1521cb3d5a7a6924e7ac6be3cec909e72ad: Status 404 returned error can't find the container with id c049962f9637a48faf4fbe6f6d69d1521cb3d5a7a6924e7ac6be3cec909e72ad Sep 12 01:58:11.108848 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:11.108812 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:11.380508 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:11.380337 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:11.513664 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:11.512797 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:11.591697 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:11.591537 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3fc028-67db4f49b5-5jjjf" event={"ID":"6651f43f-9684-460e-b2f6-787e35ebe96d","Type":"ContainerStarted","Data":"b64484c70e9d4e8cf20ef89b038f12687d0653b7d0cc78ac74d05a5ecb2f92eb"} Sep 12 01:58:11.591697 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:11.591607 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3fc028-67db4f49b5-5jjjf" event={"ID":"6651f43f-9684-460e-b2f6-787e35ebe96d","Type":"ContainerStarted","Data":"c049962f9637a48faf4fbe6f6d69d1521cb3d5a7a6924e7ac6be3cec909e72ad"} Sep 12 01:58:11.591697 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:11.591660 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-3fc028-67db4f49b5-5jjjf" Sep 12 01:58:11.597050 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:11.596971 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f660a6-7c79ccd4b6-l6sfk" event={"ID":"c273cea1-7278-4513-8432-ccba4c25c999","Type":"ContainerStarted","Data":"491e9a7275f615f2517a7ae00fda6023c4bd0e406fac9d4a0e3e89383cb1cc3c"} Sep 12 01:58:11.597353 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:11.597260 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f660a6-7c79ccd4b6-l6sfk" Sep 12 01:58:11.613242 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:11.613019 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-3fc028-67db4f49b5-5jjjf" podStartSLOduration=1.30681808 podStartE2EDuration="1.612995695s" totalImagesPullingTime="306.177615ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:58:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:58:11.609787792 +0000 UTC m=+794.401781597" watchObservedRunningTime="2026-09-12 01:58:11.612995695 +0000 UTC m=+794.404989500" Sep 12 01:58:11.876531 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:11.876484 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:12.456531 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:12.456498 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:12.525377 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:12.525331 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:12.640197 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:12.640156 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:12.743695 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:12.743580 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:12.847473 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:12.840386 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:13.053825 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:13.053737 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:13.153167 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:13.153129 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:13.398837 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:13.398802 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:13.500000 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:13.499966 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:13.610115 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:13.610027 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3fc028-668b6db875-svkb5" event={"ID":"268dd63e-c941-4687-ba08-f4e342b2585f","Type":"ContainerStarted","Data":"421b529c01a7bd5fb73cddacd7090ea69ee09be71fa25b219e5e31c7aa92712c"} Sep 12 01:58:13.610281 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:13.610183 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3fc028-668b6db875-svkb5" Sep 12 01:58:13.628084 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:13.628034 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f660a6-7c79ccd4b6-l6sfk" podStartSLOduration=3.183611988 podStartE2EDuration="4.628015999s" totalImagesPullingTime="1.444404011s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:58:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:58:11.63277471 +0000 UTC m=+794.424768506" watchObservedRunningTime="2026-09-12 01:58:13.628015999 +0000 UTC m=+796.420009797" Sep 12 01:58:13.628971 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:13.628935 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3fc028-668b6db875-svkb5" podStartSLOduration=1.890491632 podStartE2EDuration="4.628922355s" totalImagesPullingTime="2.738430723s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:58:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:58:13.626064124 +0000 UTC m=+796.418057919" watchObservedRunningTime="2026-09-12 01:58:13.628922355 +0000 UTC m=+796.420916155" Sep 12 01:58:15.167828 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:15.167793 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:15.618633 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:15.618551 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:15.716683 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:15.716649 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:16.373000 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:16.372954 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:16.475903 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:16.475872 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:16.570526 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:16.570490 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:17.776396 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:17.776368 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:18.634822 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:18.634791 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:18.834241 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:18.834160 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 01:58:19.622292 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:19.622256 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3fc028-668b6db875-svkb5" Sep 12 01:58:19.871661 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:19.871626 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-9ef13e-7f6b799448-rmszj"] Sep 12 01:58:19.876563 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:19.876493 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9ef13e-7f6b799448-rmszj" Sep 12 01:58:19.878601 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:19.878583 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-9ef13e-serving-cert\"" Sep 12 01:58:19.884102 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:19.884073 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-9ef13e-7f6b799448-rmszj"] Sep 12 01:58:19.941967 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:19.941930 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:19.994931 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:19.994894 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca57934c-35fb-4cac-8c96-e3861edbb214-maas-api-tls\") pod \"maas-api-e2e-autores-9ef13e-7f6b799448-rmszj\" (UID: \"ca57934c-35fb-4cac-8c96-e3861edbb214\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9ef13e-7f6b799448-rmszj" Sep 12 01:58:19.995136 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:19.994970 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca57934c-35fb-4cac-8c96-e3861edbb214-openshift-service-ca\") pod \"maas-api-e2e-autores-9ef13e-7f6b799448-rmszj\" (UID: \"ca57934c-35fb-4cac-8c96-e3861edbb214\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9ef13e-7f6b799448-rmszj" Sep 12 01:58:19.995136 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:19.995036 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rvmv\" (UniqueName: \"kubernetes.io/projected/ca57934c-35fb-4cac-8c96-e3861edbb214-kube-api-access-7rvmv\") pod \"maas-api-e2e-autores-9ef13e-7f6b799448-rmszj\" (UID: \"ca57934c-35fb-4cac-8c96-e3861edbb214\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9ef13e-7f6b799448-rmszj" Sep 12 01:58:19.995136 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:19.995116 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca57934c-35fb-4cac-8c96-e3861edbb214-metrics-certs\") pod \"maas-api-e2e-autores-9ef13e-7f6b799448-rmszj\" (UID: \"ca57934c-35fb-4cac-8c96-e3861edbb214\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9ef13e-7f6b799448-rmszj" Sep 12 01:58:20.041972 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:20.041929 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:20.096619 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:20.096585 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca57934c-35fb-4cac-8c96-e3861edbb214-metrics-certs\") pod \"maas-api-e2e-autores-9ef13e-7f6b799448-rmszj\" (UID: \"ca57934c-35fb-4cac-8c96-e3861edbb214\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9ef13e-7f6b799448-rmszj" Sep 12 01:58:20.096819 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:20.096630 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca57934c-35fb-4cac-8c96-e3861edbb214-maas-api-tls\") pod \"maas-api-e2e-autores-9ef13e-7f6b799448-rmszj\" (UID: \"ca57934c-35fb-4cac-8c96-e3861edbb214\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9ef13e-7f6b799448-rmszj" Sep 12 01:58:20.096819 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:20.096677 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca57934c-35fb-4cac-8c96-e3861edbb214-openshift-service-ca\") pod \"maas-api-e2e-autores-9ef13e-7f6b799448-rmszj\" (UID: \"ca57934c-35fb-4cac-8c96-e3861edbb214\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9ef13e-7f6b799448-rmszj" Sep 12 01:58:20.096819 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:20.096718 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rvmv\" (UniqueName: \"kubernetes.io/projected/ca57934c-35fb-4cac-8c96-e3861edbb214-kube-api-access-7rvmv\") pod \"maas-api-e2e-autores-9ef13e-7f6b799448-rmszj\" (UID: \"ca57934c-35fb-4cac-8c96-e3861edbb214\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9ef13e-7f6b799448-rmszj" Sep 12 01:58:20.096992 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:58:20.096820 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-9ef13e-serving-cert: secret "maas-api-e2e-autores-9ef13e-serving-cert" not found Sep 12 01:58:20.096992 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:58:20.096887 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca57934c-35fb-4cac-8c96-e3861edbb214-maas-api-tls podName:ca57934c-35fb-4cac-8c96-e3861edbb214 nodeName:}" failed. No retries permitted until 2026-09-12 01:58:20.596872211 +0000 UTC m=+803.388865997 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ca57934c-35fb-4cac-8c96-e3861edbb214-maas-api-tls") pod "maas-api-e2e-autores-9ef13e-7f6b799448-rmszj" (UID: "ca57934c-35fb-4cac-8c96-e3861edbb214") : secret "maas-api-e2e-autores-9ef13e-serving-cert" not found Sep 12 01:58:20.097375 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:20.097359 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca57934c-35fb-4cac-8c96-e3861edbb214-openshift-service-ca\") pod \"maas-api-e2e-autores-9ef13e-7f6b799448-rmszj\" (UID: \"ca57934c-35fb-4cac-8c96-e3861edbb214\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9ef13e-7f6b799448-rmszj" Sep 12 01:58:20.099328 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:20.099292 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca57934c-35fb-4cac-8c96-e3861edbb214-metrics-certs\") pod \"maas-api-e2e-autores-9ef13e-7f6b799448-rmszj\" (UID: \"ca57934c-35fb-4cac-8c96-e3861edbb214\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9ef13e-7f6b799448-rmszj" Sep 12 01:58:20.106847 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:20.106823 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rvmv\" (UniqueName: \"kubernetes.io/projected/ca57934c-35fb-4cac-8c96-e3861edbb214-kube-api-access-7rvmv\") pod \"maas-api-e2e-autores-9ef13e-7f6b799448-rmszj\" (UID: \"ca57934c-35fb-4cac-8c96-e3861edbb214\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9ef13e-7f6b799448-rmszj" Sep 12 01:58:20.159534 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:20.159412 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:20.437196 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:20.437093 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:20.535430 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:20.535396 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:20.602113 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:20.602084 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca57934c-35fb-4cac-8c96-e3861edbb214-maas-api-tls\") pod \"maas-api-e2e-autores-9ef13e-7f6b799448-rmszj\" (UID: \"ca57934c-35fb-4cac-8c96-e3861edbb214\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9ef13e-7f6b799448-rmszj" Sep 12 01:58:20.604699 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:20.604678 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca57934c-35fb-4cac-8c96-e3861edbb214-maas-api-tls\") pod \"maas-api-e2e-autores-9ef13e-7f6b799448-rmszj\" (UID: \"ca57934c-35fb-4cac-8c96-e3861edbb214\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9ef13e-7f6b799448-rmszj" Sep 12 01:58:20.787808 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:20.787719 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9ef13e-7f6b799448-rmszj" Sep 12 01:58:21.060739 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:21.060715 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-9ef13e-7f6b799448-rmszj"] Sep 12 01:58:21.063236 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:58:21.063210 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca57934c_35fb_4cac_8c96_e3861edbb214.slice/crio-0df5268f0dbf1f024625896e8baddf62d2de755912d857aa03948357a126978c WatchSource:0}: Error finding container 0df5268f0dbf1f024625896e8baddf62d2de755912d857aa03948357a126978c: Status 404 returned error can't find the container with id 0df5268f0dbf1f024625896e8baddf62d2de755912d857aa03948357a126978c Sep 12 01:58:21.126444 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:21.126410 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-57c4f4a4-bd67c5cfb-n9blb"] Sep 12 01:58:21.132068 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:21.132042 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-57c4f4a4-bd67c5cfb-n9blb" Sep 12 01:58:21.134680 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:21.134635 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-57c4f4a4-serving-cert\"" Sep 12 01:58:21.139715 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:21.139685 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-57c4f4a4-bd67c5cfb-n9blb"] Sep 12 01:58:21.160599 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:21.160565 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:21.208787 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:21.208747 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwf9w\" (UniqueName: \"kubernetes.io/projected/e3296116-4ba4-459e-acf9-bd4cfdb61996-kube-api-access-hwf9w\") pod \"maas-api-e2e-ait-57c4f4a4-bd67c5cfb-n9blb\" (UID: \"e3296116-4ba4-459e-acf9-bd4cfdb61996\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-57c4f4a4-bd67c5cfb-n9blb" Sep 12 01:58:21.208787 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:21.208788 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3296116-4ba4-459e-acf9-bd4cfdb61996-openshift-service-ca\") pod \"maas-api-e2e-ait-57c4f4a4-bd67c5cfb-n9blb\" (UID: \"e3296116-4ba4-459e-acf9-bd4cfdb61996\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-57c4f4a4-bd67c5cfb-n9blb" Sep 12 01:58:21.209014 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:21.208880 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3296116-4ba4-459e-acf9-bd4cfdb61996-maas-api-tls\") pod \"maas-api-e2e-ait-57c4f4a4-bd67c5cfb-n9blb\" (UID: \"e3296116-4ba4-459e-acf9-bd4cfdb61996\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-57c4f4a4-bd67c5cfb-n9blb" Sep 12 01:58:21.209014 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:21.208917 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3296116-4ba4-459e-acf9-bd4cfdb61996-metrics-certs\") pod \"maas-api-e2e-ait-57c4f4a4-bd67c5cfb-n9blb\" (UID: \"e3296116-4ba4-459e-acf9-bd4cfdb61996\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-57c4f4a4-bd67c5cfb-n9blb" Sep 12 01:58:21.246761 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:21.246723 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:21.310130 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:21.310094 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3296116-4ba4-459e-acf9-bd4cfdb61996-openshift-service-ca\") pod \"maas-api-e2e-ait-57c4f4a4-bd67c5cfb-n9blb\" (UID: \"e3296116-4ba4-459e-acf9-bd4cfdb61996\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-57c4f4a4-bd67c5cfb-n9blb" Sep 12 01:58:21.310327 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:21.310170 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3296116-4ba4-459e-acf9-bd4cfdb61996-maas-api-tls\") pod \"maas-api-e2e-ait-57c4f4a4-bd67c5cfb-n9blb\" (UID: \"e3296116-4ba4-459e-acf9-bd4cfdb61996\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-57c4f4a4-bd67c5cfb-n9blb" Sep 12 01:58:21.310327 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:21.310208 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3296116-4ba4-459e-acf9-bd4cfdb61996-metrics-certs\") pod \"maas-api-e2e-ait-57c4f4a4-bd67c5cfb-n9blb\" (UID: \"e3296116-4ba4-459e-acf9-bd4cfdb61996\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-57c4f4a4-bd67c5cfb-n9blb" Sep 12 01:58:21.310649 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:21.310335 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwf9w\" (UniqueName: \"kubernetes.io/projected/e3296116-4ba4-459e-acf9-bd4cfdb61996-kube-api-access-hwf9w\") pod \"maas-api-e2e-ait-57c4f4a4-bd67c5cfb-n9blb\" (UID: \"e3296116-4ba4-459e-acf9-bd4cfdb61996\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-57c4f4a4-bd67c5cfb-n9blb" Sep 12 01:58:21.310649 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:58:21.310343 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-57c4f4a4-serving-cert: secret "maas-api-e2e-ait-57c4f4a4-serving-cert" not found Sep 12 01:58:21.310649 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:58:21.310425 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3296116-4ba4-459e-acf9-bd4cfdb61996-maas-api-tls podName:e3296116-4ba4-459e-acf9-bd4cfdb61996 nodeName:}" failed. No retries permitted until 2026-09-12 01:58:21.810402872 +0000 UTC m=+804.602396659 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e3296116-4ba4-459e-acf9-bd4cfdb61996-maas-api-tls") pod "maas-api-e2e-ait-57c4f4a4-bd67c5cfb-n9blb" (UID: "e3296116-4ba4-459e-acf9-bd4cfdb61996") : secret "maas-api-e2e-ait-57c4f4a4-serving-cert" not found Sep 12 01:58:21.310966 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:21.310945 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3296116-4ba4-459e-acf9-bd4cfdb61996-openshift-service-ca\") pod \"maas-api-e2e-ait-57c4f4a4-bd67c5cfb-n9blb\" (UID: \"e3296116-4ba4-459e-acf9-bd4cfdb61996\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-57c4f4a4-bd67c5cfb-n9blb" Sep 12 01:58:21.312982 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:21.312955 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3296116-4ba4-459e-acf9-bd4cfdb61996-metrics-certs\") pod \"maas-api-e2e-ait-57c4f4a4-bd67c5cfb-n9blb\" (UID: \"e3296116-4ba4-459e-acf9-bd4cfdb61996\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-57c4f4a4-bd67c5cfb-n9blb" Sep 12 01:58:21.321488 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:21.321441 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwf9w\" (UniqueName: \"kubernetes.io/projected/e3296116-4ba4-459e-acf9-bd4cfdb61996-kube-api-access-hwf9w\") pod \"maas-api-e2e-ait-57c4f4a4-bd67c5cfb-n9blb\" (UID: \"e3296116-4ba4-459e-acf9-bd4cfdb61996\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-57c4f4a4-bd67c5cfb-n9blb" Sep 12 01:58:21.549310 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:21.549264 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:21.651926 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:21.651892 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:21.652135 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:21.652110 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9ef13e-7f6b799448-rmszj" event={"ID":"ca57934c-35fb-4cac-8c96-e3861edbb214","Type":"ContainerStarted","Data":"6c15699cc377180d2a6cccae0cac8152d2086b39b4878db3f8351ceb39bfa926"} Sep 12 01:58:21.652225 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:21.652209 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9ef13e-7f6b799448-rmszj" event={"ID":"ca57934c-35fb-4cac-8c96-e3861edbb214","Type":"ContainerStarted","Data":"0df5268f0dbf1f024625896e8baddf62d2de755912d857aa03948357a126978c"} Sep 12 01:58:21.652290 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:21.652278 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9ef13e-7f6b799448-rmszj" Sep 12 01:58:21.678645 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:21.678595 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9ef13e-7f6b799448-rmszj" podStartSLOduration=2.362954742 podStartE2EDuration="2.678577139s" totalImagesPullingTime="315.622397ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:58:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:58:21.676679843 +0000 UTC m=+804.468673656" watchObservedRunningTime="2026-09-12 01:58:21.678577139 +0000 UTC m=+804.470570926" Sep 12 01:58:21.753014 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:21.752978 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:21.816412 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:21.816373 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3296116-4ba4-459e-acf9-bd4cfdb61996-maas-api-tls\") pod \"maas-api-e2e-ait-57c4f4a4-bd67c5cfb-n9blb\" (UID: \"e3296116-4ba4-459e-acf9-bd4cfdb61996\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-57c4f4a4-bd67c5cfb-n9blb" Sep 12 01:58:21.819542 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:21.819511 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3296116-4ba4-459e-acf9-bd4cfdb61996-maas-api-tls\") pod \"maas-api-e2e-ait-57c4f4a4-bd67c5cfb-n9blb\" (UID: \"e3296116-4ba4-459e-acf9-bd4cfdb61996\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-57c4f4a4-bd67c5cfb-n9blb" Sep 12 01:58:22.046721 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:22.046625 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-57c4f4a4-bd67c5cfb-n9blb" Sep 12 01:58:22.081683 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:22.081647 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:22.172165 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:22.172120 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:22.303811 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:22.303730 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:22.323057 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:22.323034 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-57c4f4a4-bd67c5cfb-n9blb"] Sep 12 01:58:22.325368 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:58:22.325336 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3296116_4ba4_459e_acf9_bd4cfdb61996.slice/crio-6f75d68a9fa355f8ffc27c3fabce1553354e313bf56d6d60b0defc5090baa39e WatchSource:0}: Error finding container 6f75d68a9fa355f8ffc27c3fabce1553354e313bf56d6d60b0defc5090baa39e: Status 404 returned error can't find the container with id 6f75d68a9fa355f8ffc27c3fabce1553354e313bf56d6d60b0defc5090baa39e Sep 12 01:58:22.606019 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:22.605928 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-3fc028-67db4f49b5-5jjjf" Sep 12 01:58:22.606019 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:22.606017 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f660a6-7c79ccd4b6-l6sfk" Sep 12 01:58:22.660519 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:22.660482 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-57c4f4a4-bd67c5cfb-n9blb" event={"ID":"e3296116-4ba4-459e-acf9-bd4cfdb61996","Type":"ContainerStarted","Data":"6f75d68a9fa355f8ffc27c3fabce1553354e313bf56d6d60b0defc5090baa39e"} Sep 12 01:58:23.337042 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:23.337005 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:23.668535 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:23.668491 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-57c4f4a4-bd67c5cfb-n9blb" event={"ID":"e3296116-4ba4-459e-acf9-bd4cfdb61996","Type":"ContainerStarted","Data":"ed2000307ecb84988945d410b72bcf0ce7810ddc6e73cecbee84e0df53a1a09f"} Sep 12 01:58:23.668680 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:23.668626 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-57c4f4a4-bd67c5cfb-n9blb" Sep 12 01:58:23.688362 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:23.688308 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-57c4f4a4-bd67c5cfb-n9blb" podStartSLOduration=2.4008555080000002 podStartE2EDuration="2.688291473s" totalImagesPullingTime="287.435965ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:58:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:58:23.686714441 +0000 UTC m=+806.478708236" watchObservedRunningTime="2026-09-12 01:58:23.688291473 +0000 UTC m=+806.480285266" Sep 12 01:58:24.033548 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:24.033437 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:24.204443 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:24.204399 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:24.332742 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:24.332664 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:24.431664 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:24.431632 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:24.541154 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:24.541121 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:24.636383 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:24.636348 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:24.875744 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:24.875701 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:25.586566 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:25.586530 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:26.410093 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:26.410057 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:26.702440 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:26.702351 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:27.036054 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:27.035956 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:27.131304 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:27.131265 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:27.230802 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:27.230765 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:27.435929 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:27.435894 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:27.531401 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:27.531372 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:27.627548 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:27.627512 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:27.666810 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:27.666785 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9ef13e-7f6b799448-rmszj" Sep 12 01:58:27.780865 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:27.780787 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:27.932796 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:27.932758 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:28.878834 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:28.878795 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:29.050885 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:29.050841 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:29.678783 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:29.678753 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-57c4f4a4-bd67c5cfb-n9blb" Sep 12 01:58:29.926193 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:29.926154 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:30.408356 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:30.408315 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:30.506352 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:30.506318 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:31.049661 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:31.049631 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:31.499600 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:31.499565 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:31.617859 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:31.617804 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:31.804543 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:31.804433 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:31.911440 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:31.911410 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:32.007510 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:32.007422 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:32.114389 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:32.114300 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:32.422678 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:32.422643 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:32.524286 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:32.524250 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:32.543227 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:32.543196 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7d6e9f-758b6b6f8b-hdf5b"] Sep 12 01:58:32.547052 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:32.547032 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7d6e9f-758b6b6f8b-hdf5b" Sep 12 01:58:32.550531 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:32.550507 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-7d6e9f-dockercfg-gvcvf\"" Sep 12 01:58:32.551264 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:32.551237 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-7d6e9f\"" Sep 12 01:58:32.560009 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:32.559985 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7d6e9f-758b6b6f8b-hdf5b"] Sep 12 01:58:32.724893 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:32.724791 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsmwr\" (UniqueName: \"kubernetes.io/projected/b3213bb0-34c6-4614-9da9-64119a400de7-kube-api-access-hsmwr\") pod \"payload-pre-processing-e2e-inf-b-7d6e9f-758b6b6f8b-hdf5b\" (UID: \"b3213bb0-34c6-4614-9da9-64119a400de7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7d6e9f-758b6b6f8b-hdf5b" Sep 12 01:58:32.725062 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:32.724890 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3213bb0-34c6-4614-9da9-64119a400de7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-7d6e9f-758b6b6f8b-hdf5b\" (UID: \"b3213bb0-34c6-4614-9da9-64119a400de7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7d6e9f-758b6b6f8b-hdf5b" Sep 12 01:58:32.826347 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:32.826308 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsmwr\" (UniqueName: \"kubernetes.io/projected/b3213bb0-34c6-4614-9da9-64119a400de7-kube-api-access-hsmwr\") pod \"payload-pre-processing-e2e-inf-b-7d6e9f-758b6b6f8b-hdf5b\" (UID: \"b3213bb0-34c6-4614-9da9-64119a400de7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7d6e9f-758b6b6f8b-hdf5b" Sep 12 01:58:32.826529 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:32.826376 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3213bb0-34c6-4614-9da9-64119a400de7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-7d6e9f-758b6b6f8b-hdf5b\" (UID: \"b3213bb0-34c6-4614-9da9-64119a400de7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7d6e9f-758b6b6f8b-hdf5b" Sep 12 01:58:32.827071 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:32.827052 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3213bb0-34c6-4614-9da9-64119a400de7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-7d6e9f-758b6b6f8b-hdf5b\" (UID: \"b3213bb0-34c6-4614-9da9-64119a400de7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7d6e9f-758b6b6f8b-hdf5b" Sep 12 01:58:32.835852 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:32.835830 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsmwr\" (UniqueName: \"kubernetes.io/projected/b3213bb0-34c6-4614-9da9-64119a400de7-kube-api-access-hsmwr\") pod \"payload-pre-processing-e2e-inf-b-7d6e9f-758b6b6f8b-hdf5b\" (UID: \"b3213bb0-34c6-4614-9da9-64119a400de7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7d6e9f-758b6b6f8b-hdf5b" Sep 12 01:58:32.858408 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:32.858373 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7d6e9f-758b6b6f8b-hdf5b" Sep 12 01:58:33.001414 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:33.001331 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:33.132592 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:33.132568 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7d6e9f-758b6b6f8b-hdf5b"] Sep 12 01:58:33.135125 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:58:33.135098 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3213bb0_34c6_4614_9da9_64119a400de7.slice/crio-860ba7f1d276fc094a1afeb4c06becd643949156d389d996188485ded7d028ca WatchSource:0}: Error finding container 860ba7f1d276fc094a1afeb4c06becd643949156d389d996188485ded7d028ca: Status 404 returned error can't find the container with id 860ba7f1d276fc094a1afeb4c06becd643949156d389d996188485ded7d028ca Sep 12 01:58:33.729158 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:33.729123 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7d6e9f-758b6b6f8b-hdf5b" event={"ID":"b3213bb0-34c6-4614-9da9-64119a400de7","Type":"ContainerStarted","Data":"460cf300a373fe6d4c576eaa87b116076752d2e523953a9a707063a9e5c7e539"} Sep 12 01:58:33.729337 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:33.729197 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7d6e9f-758b6b6f8b-hdf5b" Sep 12 01:58:33.729337 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:33.729222 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7d6e9f-758b6b6f8b-hdf5b" event={"ID":"b3213bb0-34c6-4614-9da9-64119a400de7","Type":"ContainerStarted","Data":"860ba7f1d276fc094a1afeb4c06becd643949156d389d996188485ded7d028ca"} Sep 12 01:58:33.741915 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:33.741864 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7d6e9f-758b6b6f8b-hdf5b" podStartSLOduration=1.741850977 podStartE2EDuration="1.741850977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:58:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:58:33.740216258 +0000 UTC m=+816.532210053" watchObservedRunningTime="2026-09-12 01:58:33.741850977 +0000 UTC m=+816.533844771" Sep 12 01:58:33.753429 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:33.753391 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:34.339262 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:34.339229 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 01:58:34.670614 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:34.670581 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:35.543472 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:35.543364 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:36.137609 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:36.137571 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:36.336752 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:36.336717 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:37.170588 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:37.170551 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:37.909508 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:37.909465 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:38.639856 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:38.639819 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:38.739359 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:38.739326 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:38.841338 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:38.841286 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:39.438569 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:39.438534 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:40.158544 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:40.158510 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:40.895643 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:40.895611 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:41.584832 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:41.584801 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:42.363656 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:42.363627 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:43.003175 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:43.003140 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:44.730375 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:44.730346 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7d6e9f-758b6b6f8b-hdf5b" Sep 12 01:58:46.584672 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:46.584640 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-57c4f4a4-bd67c5cfb-n9blb"] Sep 12 01:58:46.585077 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:46.584901 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-57c4f4a4-bd67c5cfb-n9blb" podUID="e3296116-4ba4-459e-acf9-bd4cfdb61996" containerName="maas-api" containerID="cri-o://ed2000307ecb84988945d410b72bcf0ce7810ddc6e73cecbee84e0df53a1a09f" gracePeriod=30 Sep 12 01:58:46.608708 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:46.608676 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:46.686365 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:46.686329 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:46.784580 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:46.784536 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:46.791789 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:46.791755 2579 generic.go:356] "Generic (PLEG): container finished" podID="e3296116-4ba4-459e-acf9-bd4cfdb61996" containerID="ed2000307ecb84988945d410b72bcf0ce7810ddc6e73cecbee84e0df53a1a09f" exitCode=0 Sep 12 01:58:46.791972 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:46.791801 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-57c4f4a4-bd67c5cfb-n9blb" event={"ID":"e3296116-4ba4-459e-acf9-bd4cfdb61996","Type":"ContainerDied","Data":"ed2000307ecb84988945d410b72bcf0ce7810ddc6e73cecbee84e0df53a1a09f"} Sep 12 01:58:46.842588 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:46.842527 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-57c4f4a4-bd67c5cfb-n9blb" Sep 12 01:58:46.863239 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:46.863140 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:46.926175 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:46.926121 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:46.969564 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:46.969533 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3296116-4ba4-459e-acf9-bd4cfdb61996-metrics-certs\") pod \"e3296116-4ba4-459e-acf9-bd4cfdb61996\" (UID: \"e3296116-4ba4-459e-acf9-bd4cfdb61996\") " Sep 12 01:58:46.969564 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:46.969566 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3296116-4ba4-459e-acf9-bd4cfdb61996-maas-api-tls\") pod \"e3296116-4ba4-459e-acf9-bd4cfdb61996\" (UID: \"e3296116-4ba4-459e-acf9-bd4cfdb61996\") " Sep 12 01:58:46.969876 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:46.969622 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3296116-4ba4-459e-acf9-bd4cfdb61996-openshift-service-ca\") pod \"e3296116-4ba4-459e-acf9-bd4cfdb61996\" (UID: \"e3296116-4ba4-459e-acf9-bd4cfdb61996\") " Sep 12 01:58:46.969876 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:46.969676 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hwf9w\" (UniqueName: \"kubernetes.io/projected/e3296116-4ba4-459e-acf9-bd4cfdb61996-kube-api-access-hwf9w\") pod \"e3296116-4ba4-459e-acf9-bd4cfdb61996\" (UID: \"e3296116-4ba4-459e-acf9-bd4cfdb61996\") " Sep 12 01:58:46.970000 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:46.969981 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3296116-4ba4-459e-acf9-bd4cfdb61996-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e3296116-4ba4-459e-acf9-bd4cfdb61996" (UID: "e3296116-4ba4-459e-acf9-bd4cfdb61996"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 12 01:58:46.972044 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:46.972014 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3296116-4ba4-459e-acf9-bd4cfdb61996-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e3296116-4ba4-459e-acf9-bd4cfdb61996" (UID: "e3296116-4ba4-459e-acf9-bd4cfdb61996"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 12 01:58:46.972044 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:46.972021 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3296116-4ba4-459e-acf9-bd4cfdb61996-kube-api-access-hwf9w" (OuterVolumeSpecName: "kube-api-access-hwf9w") pod "e3296116-4ba4-459e-acf9-bd4cfdb61996" (UID: "e3296116-4ba4-459e-acf9-bd4cfdb61996"). InnerVolumeSpecName "kube-api-access-hwf9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 12 01:58:46.972267 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:46.972040 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3296116-4ba4-459e-acf9-bd4cfdb61996-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e3296116-4ba4-459e-acf9-bd4cfdb61996" (UID: "e3296116-4ba4-459e-acf9-bd4cfdb61996"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 12 01:58:47.031964 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:47.031920 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:47.071533 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:47.071496 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3296116-4ba4-459e-acf9-bd4cfdb61996-openshift-service-ca\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 01:58:47.071533 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:47.071526 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwf9w\" (UniqueName: \"kubernetes.io/projected/e3296116-4ba4-459e-acf9-bd4cfdb61996-kube-api-access-hwf9w\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 01:58:47.071533 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:47.071542 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3296116-4ba4-459e-acf9-bd4cfdb61996-metrics-certs\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 01:58:47.071818 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:47.071552 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3296116-4ba4-459e-acf9-bd4cfdb61996-maas-api-tls\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 01:58:47.241054 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:47.241016 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:47.500708 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:47.500594 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:47.592556 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:47.592510 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:47.765472 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:47.765347 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:47.798516 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:47.798483 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-57c4f4a4-bd67c5cfb-n9blb" event={"ID":"e3296116-4ba4-459e-acf9-bd4cfdb61996","Type":"ContainerDied","Data":"6f75d68a9fa355f8ffc27c3fabce1553354e313bf56d6d60b0defc5090baa39e"} Sep 12 01:58:47.798516 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:47.798516 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-57c4f4a4-bd67c5cfb-n9blb" Sep 12 01:58:47.798740 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:47.798570 2579 scope.go:117] "RemoveContainer" containerID="ed2000307ecb84988945d410b72bcf0ce7810ddc6e73cecbee84e0df53a1a09f" Sep 12 01:58:47.828500 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:47.828430 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-57c4f4a4-bd67c5cfb-n9blb"] Sep 12 01:58:47.834992 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:47.834953 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-57c4f4a4-bd67c5cfb-n9blb"] Sep 12 01:58:48.249957 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:48.249925 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:48.351414 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:48.351362 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:48.510495 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:48.510361 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:49.329704 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:49.329675 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:49.726838 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:49.726794 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3296116-4ba4-459e-acf9-bd4cfdb61996" path="/var/lib/kubelet/pods/e3296116-4ba4-459e-acf9-bd4cfdb61996/volumes" Sep 12 01:58:50.042099 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:50.042004 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:50.320599 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:50.320500 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:50.933781 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:50.933742 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:51.035583 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:51.035541 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:51.134084 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:51.134033 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:51.715653 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:51.715622 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:52.576954 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:52.576921 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:53.261467 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:53.261427 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:53.938148 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:53.938118 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 01:58:54.122017 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:54.121971 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:54.869795 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:54.869764 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-899a9ad9-666bbd9985-99dz9"] Sep 12 01:58:54.870631 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:54.870380 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3296116-4ba4-459e-acf9-bd4cfdb61996" containerName="maas-api" Sep 12 01:58:54.870631 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:54.870408 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3296116-4ba4-459e-acf9-bd4cfdb61996" containerName="maas-api" Sep 12 01:58:54.870631 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:54.870600 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3296116-4ba4-459e-acf9-bd4cfdb61996" containerName="maas-api" Sep 12 01:58:54.872980 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:54.872959 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-899a9ad9-666bbd9985-99dz9" Sep 12 01:58:54.875764 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:54.875740 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-899a9ad9-serving-cert\"" Sep 12 01:58:54.885140 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:54.885096 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-899a9ad9-666bbd9985-99dz9"] Sep 12 01:58:54.907477 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:54.907430 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:54.949292 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:54.949253 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48vmm\" (UniqueName: \"kubernetes.io/projected/a5f01764-9f05-4ed2-8a14-9100827327fa-kube-api-access-48vmm\") pod \"maas-api-e2e-migrate-899a9ad9-666bbd9985-99dz9\" (UID: \"a5f01764-9f05-4ed2-8a14-9100827327fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-899a9ad9-666bbd9985-99dz9" Sep 12 01:58:54.949292 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:54.949297 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5f01764-9f05-4ed2-8a14-9100827327fa-maas-api-tls\") pod \"maas-api-e2e-migrate-899a9ad9-666bbd9985-99dz9\" (UID: \"a5f01764-9f05-4ed2-8a14-9100827327fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-899a9ad9-666bbd9985-99dz9" Sep 12 01:58:54.949740 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:54.949319 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5f01764-9f05-4ed2-8a14-9100827327fa-metrics-certs\") pod \"maas-api-e2e-migrate-899a9ad9-666bbd9985-99dz9\" (UID: \"a5f01764-9f05-4ed2-8a14-9100827327fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-899a9ad9-666bbd9985-99dz9" Sep 12 01:58:54.949740 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:54.949436 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5f01764-9f05-4ed2-8a14-9100827327fa-openshift-service-ca\") pod \"maas-api-e2e-migrate-899a9ad9-666bbd9985-99dz9\" (UID: \"a5f01764-9f05-4ed2-8a14-9100827327fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-899a9ad9-666bbd9985-99dz9" Sep 12 01:58:55.007733 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:55.007699 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:55.050353 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:55.050258 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48vmm\" (UniqueName: \"kubernetes.io/projected/a5f01764-9f05-4ed2-8a14-9100827327fa-kube-api-access-48vmm\") pod \"maas-api-e2e-migrate-899a9ad9-666bbd9985-99dz9\" (UID: \"a5f01764-9f05-4ed2-8a14-9100827327fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-899a9ad9-666bbd9985-99dz9" Sep 12 01:58:55.050607 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:55.050400 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5f01764-9f05-4ed2-8a14-9100827327fa-maas-api-tls\") pod \"maas-api-e2e-migrate-899a9ad9-666bbd9985-99dz9\" (UID: \"a5f01764-9f05-4ed2-8a14-9100827327fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-899a9ad9-666bbd9985-99dz9" Sep 12 01:58:55.050607 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:55.050419 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5f01764-9f05-4ed2-8a14-9100827327fa-metrics-certs\") pod \"maas-api-e2e-migrate-899a9ad9-666bbd9985-99dz9\" (UID: \"a5f01764-9f05-4ed2-8a14-9100827327fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-899a9ad9-666bbd9985-99dz9" Sep 12 01:58:55.050607 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:55.050530 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5f01764-9f05-4ed2-8a14-9100827327fa-openshift-service-ca\") pod \"maas-api-e2e-migrate-899a9ad9-666bbd9985-99dz9\" (UID: \"a5f01764-9f05-4ed2-8a14-9100827327fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-899a9ad9-666bbd9985-99dz9" Sep 12 01:58:55.050607 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:58:55.050561 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-899a9ad9-serving-cert: secret "maas-api-e2e-migrate-899a9ad9-serving-cert" not found Sep 12 01:58:55.050847 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:58:55.050643 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5f01764-9f05-4ed2-8a14-9100827327fa-maas-api-tls podName:a5f01764-9f05-4ed2-8a14-9100827327fa nodeName:}" failed. No retries permitted until 2026-09-12 01:58:55.550620092 +0000 UTC m=+838.342613886 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a5f01764-9f05-4ed2-8a14-9100827327fa-maas-api-tls") pod "maas-api-e2e-migrate-899a9ad9-666bbd9985-99dz9" (UID: "a5f01764-9f05-4ed2-8a14-9100827327fa") : secret "maas-api-e2e-migrate-899a9ad9-serving-cert" not found Sep 12 01:58:55.051148 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:55.051130 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5f01764-9f05-4ed2-8a14-9100827327fa-openshift-service-ca\") pod \"maas-api-e2e-migrate-899a9ad9-666bbd9985-99dz9\" (UID: \"a5f01764-9f05-4ed2-8a14-9100827327fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-899a9ad9-666bbd9985-99dz9" Sep 12 01:58:55.052943 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:55.052927 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5f01764-9f05-4ed2-8a14-9100827327fa-metrics-certs\") pod \"maas-api-e2e-migrate-899a9ad9-666bbd9985-99dz9\" (UID: \"a5f01764-9f05-4ed2-8a14-9100827327fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-899a9ad9-666bbd9985-99dz9" Sep 12 01:58:55.062305 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:55.062284 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48vmm\" (UniqueName: \"kubernetes.io/projected/a5f01764-9f05-4ed2-8a14-9100827327fa-kube-api-access-48vmm\") pod \"maas-api-e2e-migrate-899a9ad9-666bbd9985-99dz9\" (UID: \"a5f01764-9f05-4ed2-8a14-9100827327fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-899a9ad9-666bbd9985-99dz9" Sep 12 01:58:55.150014 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:55.149974 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:55.357490 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:55.357442 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:55.458977 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:55.458877 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:55.555122 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:55.555083 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5f01764-9f05-4ed2-8a14-9100827327fa-maas-api-tls\") pod \"maas-api-e2e-migrate-899a9ad9-666bbd9985-99dz9\" (UID: \"a5f01764-9f05-4ed2-8a14-9100827327fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-899a9ad9-666bbd9985-99dz9" Sep 12 01:58:55.557787 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:55.557756 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5f01764-9f05-4ed2-8a14-9100827327fa-maas-api-tls\") pod \"maas-api-e2e-migrate-899a9ad9-666bbd9985-99dz9\" (UID: \"a5f01764-9f05-4ed2-8a14-9100827327fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-899a9ad9-666bbd9985-99dz9" Sep 12 01:58:55.774157 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:55.774052 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:55.786602 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:55.786580 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-899a9ad9-666bbd9985-99dz9" Sep 12 01:58:56.063800 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:56.063777 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-899a9ad9-666bbd9985-99dz9"] Sep 12 01:58:56.066700 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:58:56.066672 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5f01764_9f05_4ed2_8a14_9100827327fa.slice/crio-b2d28af4419b65bb4158de1eb405d4dd73bbb3d8e36f54931382f3b540c1f25d WatchSource:0}: Error finding container b2d28af4419b65bb4158de1eb405d4dd73bbb3d8e36f54931382f3b540c1f25d: Status 404 returned error can't find the container with id b2d28af4419b65bb4158de1eb405d4dd73bbb3d8e36f54931382f3b540c1f25d Sep 12 01:58:56.670248 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:56.670146 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:56.853018 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:56.852983 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-899a9ad9-666bbd9985-99dz9" event={"ID":"a5f01764-9f05-4ed2-8a14-9100827327fa","Type":"ContainerStarted","Data":"0d9723d96e6f69e16d87176eec617b8c98a33368532cad416fd3797b027da874"} Sep 12 01:58:56.853205 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:56.853043 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-899a9ad9-666bbd9985-99dz9" event={"ID":"a5f01764-9f05-4ed2-8a14-9100827327fa","Type":"ContainerStarted","Data":"b2d28af4419b65bb4158de1eb405d4dd73bbb3d8e36f54931382f3b540c1f25d"} Sep 12 01:58:56.853205 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:56.853090 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-899a9ad9-666bbd9985-99dz9" Sep 12 01:58:56.876661 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:56.876604 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-899a9ad9-666bbd9985-99dz9" podStartSLOduration=2.535262521 podStartE2EDuration="2.876582233s" totalImagesPullingTime="341.319712ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:58:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:58:56.874413623 +0000 UTC m=+839.666407423" watchObservedRunningTime="2026-09-12 01:58:56.876582233 +0000 UTC m=+839.668576029" Sep 12 01:58:57.350102 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:57.350061 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:58:57.453565 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:58:57.453533 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:59:02.863625 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:02.863594 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-899a9ad9-666bbd9985-99dz9" Sep 12 01:59:03.610221 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:03.610190 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f660a6/simulator-w0-f660a6-kserve-779fbfd878-4wgwt"] Sep 12 01:59:03.614386 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:03.614364 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f660a6/simulator-w0-f660a6-kserve-779fbfd878-4wgwt" Sep 12 01:59:03.617058 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:03.617036 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f660a6\"/\"default-dockercfg-sh8kw\"" Sep 12 01:59:03.617986 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:03.617965 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f660a6\"/\"kube-root-ca.crt\"" Sep 12 01:59:03.618090 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:03.618044 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f660a6\"/\"openshift-service-ca.crt\"" Sep 12 01:59:03.618090 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:03.618053 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f660a6\"/\"simulator-w0-f660a6-kserve-self-signed-certs\"" Sep 12 01:59:03.624319 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:03.624297 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f660a6/simulator-w0-f660a6-kserve-779fbfd878-4wgwt"] Sep 12 01:59:03.734368 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:03.734328 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18093353-67b9-4a8b-9b50-b07ff688f925-model-cache\") pod \"simulator-w0-f660a6-kserve-779fbfd878-4wgwt\" (UID: \"18093353-67b9-4a8b-9b50-b07ff688f925\") " pod="e2e-models-e2e-worker-w0-f660a6/simulator-w0-f660a6-kserve-779fbfd878-4wgwt" Sep 12 01:59:03.734561 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:03.734400 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18093353-67b9-4a8b-9b50-b07ff688f925-dshm\") pod \"simulator-w0-f660a6-kserve-779fbfd878-4wgwt\" (UID: \"18093353-67b9-4a8b-9b50-b07ff688f925\") " pod="e2e-models-e2e-worker-w0-f660a6/simulator-w0-f660a6-kserve-779fbfd878-4wgwt" Sep 12 01:59:03.734561 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:03.734428 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18093353-67b9-4a8b-9b50-b07ff688f925-tmp-dir\") pod \"simulator-w0-f660a6-kserve-779fbfd878-4wgwt\" (UID: \"18093353-67b9-4a8b-9b50-b07ff688f925\") " pod="e2e-models-e2e-worker-w0-f660a6/simulator-w0-f660a6-kserve-779fbfd878-4wgwt" Sep 12 01:59:03.734702 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:03.734548 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xnkt\" (UniqueName: \"kubernetes.io/projected/18093353-67b9-4a8b-9b50-b07ff688f925-kube-api-access-7xnkt\") pod \"simulator-w0-f660a6-kserve-779fbfd878-4wgwt\" (UID: \"18093353-67b9-4a8b-9b50-b07ff688f925\") " pod="e2e-models-e2e-worker-w0-f660a6/simulator-w0-f660a6-kserve-779fbfd878-4wgwt" Sep 12 01:59:03.734702 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:03.734614 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18093353-67b9-4a8b-9b50-b07ff688f925-home\") pod \"simulator-w0-f660a6-kserve-779fbfd878-4wgwt\" (UID: \"18093353-67b9-4a8b-9b50-b07ff688f925\") " pod="e2e-models-e2e-worker-w0-f660a6/simulator-w0-f660a6-kserve-779fbfd878-4wgwt" Sep 12 01:59:03.734702 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:03.734642 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18093353-67b9-4a8b-9b50-b07ff688f925-tls-certs\") pod \"simulator-w0-f660a6-kserve-779fbfd878-4wgwt\" (UID: \"18093353-67b9-4a8b-9b50-b07ff688f925\") " pod="e2e-models-e2e-worker-w0-f660a6/simulator-w0-f660a6-kserve-779fbfd878-4wgwt" Sep 12 01:59:03.818645 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:03.818608 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:59:03.835398 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:03.835368 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18093353-67b9-4a8b-9b50-b07ff688f925-dshm\") pod \"simulator-w0-f660a6-kserve-779fbfd878-4wgwt\" (UID: \"18093353-67b9-4a8b-9b50-b07ff688f925\") " pod="e2e-models-e2e-worker-w0-f660a6/simulator-w0-f660a6-kserve-779fbfd878-4wgwt" Sep 12 01:59:03.835617 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:03.835437 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18093353-67b9-4a8b-9b50-b07ff688f925-tmp-dir\") pod \"simulator-w0-f660a6-kserve-779fbfd878-4wgwt\" (UID: \"18093353-67b9-4a8b-9b50-b07ff688f925\") " pod="e2e-models-e2e-worker-w0-f660a6/simulator-w0-f660a6-kserve-779fbfd878-4wgwt" Sep 12 01:59:03.835845 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:03.835819 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xnkt\" (UniqueName: \"kubernetes.io/projected/18093353-67b9-4a8b-9b50-b07ff688f925-kube-api-access-7xnkt\") pod \"simulator-w0-f660a6-kserve-779fbfd878-4wgwt\" (UID: \"18093353-67b9-4a8b-9b50-b07ff688f925\") " pod="e2e-models-e2e-worker-w0-f660a6/simulator-w0-f660a6-kserve-779fbfd878-4wgwt" Sep 12 01:59:03.836062 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:03.836045 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18093353-67b9-4a8b-9b50-b07ff688f925-home\") pod \"simulator-w0-f660a6-kserve-779fbfd878-4wgwt\" (UID: \"18093353-67b9-4a8b-9b50-b07ff688f925\") " pod="e2e-models-e2e-worker-w0-f660a6/simulator-w0-f660a6-kserve-779fbfd878-4wgwt" Sep 12 01:59:03.836238 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:03.835881 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18093353-67b9-4a8b-9b50-b07ff688f925-tmp-dir\") pod \"simulator-w0-f660a6-kserve-779fbfd878-4wgwt\" (UID: \"18093353-67b9-4a8b-9b50-b07ff688f925\") " pod="e2e-models-e2e-worker-w0-f660a6/simulator-w0-f660a6-kserve-779fbfd878-4wgwt" Sep 12 01:59:03.836238 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:03.836207 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18093353-67b9-4a8b-9b50-b07ff688f925-tls-certs\") pod \"simulator-w0-f660a6-kserve-779fbfd878-4wgwt\" (UID: \"18093353-67b9-4a8b-9b50-b07ff688f925\") " pod="e2e-models-e2e-worker-w0-f660a6/simulator-w0-f660a6-kserve-779fbfd878-4wgwt" Sep 12 01:59:03.836404 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:03.836274 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18093353-67b9-4a8b-9b50-b07ff688f925-model-cache\") pod \"simulator-w0-f660a6-kserve-779fbfd878-4wgwt\" (UID: \"18093353-67b9-4a8b-9b50-b07ff688f925\") " pod="e2e-models-e2e-worker-w0-f660a6/simulator-w0-f660a6-kserve-779fbfd878-4wgwt" Sep 12 01:59:03.836571 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:03.836542 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18093353-67b9-4a8b-9b50-b07ff688f925-model-cache\") pod \"simulator-w0-f660a6-kserve-779fbfd878-4wgwt\" (UID: \"18093353-67b9-4a8b-9b50-b07ff688f925\") " pod="e2e-models-e2e-worker-w0-f660a6/simulator-w0-f660a6-kserve-779fbfd878-4wgwt" Sep 12 01:59:03.836571 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:03.836546 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18093353-67b9-4a8b-9b50-b07ff688f925-home\") pod \"simulator-w0-f660a6-kserve-779fbfd878-4wgwt\" (UID: \"18093353-67b9-4a8b-9b50-b07ff688f925\") " pod="e2e-models-e2e-worker-w0-f660a6/simulator-w0-f660a6-kserve-779fbfd878-4wgwt" Sep 12 01:59:03.838040 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:03.838023 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18093353-67b9-4a8b-9b50-b07ff688f925-dshm\") pod \"simulator-w0-f660a6-kserve-779fbfd878-4wgwt\" (UID: \"18093353-67b9-4a8b-9b50-b07ff688f925\") " pod="e2e-models-e2e-worker-w0-f660a6/simulator-w0-f660a6-kserve-779fbfd878-4wgwt" Sep 12 01:59:03.838760 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:03.838744 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18093353-67b9-4a8b-9b50-b07ff688f925-tls-certs\") pod \"simulator-w0-f660a6-kserve-779fbfd878-4wgwt\" (UID: \"18093353-67b9-4a8b-9b50-b07ff688f925\") " pod="e2e-models-e2e-worker-w0-f660a6/simulator-w0-f660a6-kserve-779fbfd878-4wgwt" Sep 12 01:59:03.843824 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:03.843799 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xnkt\" (UniqueName: \"kubernetes.io/projected/18093353-67b9-4a8b-9b50-b07ff688f925-kube-api-access-7xnkt\") pod \"simulator-w0-f660a6-kserve-779fbfd878-4wgwt\" (UID: \"18093353-67b9-4a8b-9b50-b07ff688f925\") " pod="e2e-models-e2e-worker-w0-f660a6/simulator-w0-f660a6-kserve-779fbfd878-4wgwt" Sep 12 01:59:03.924764 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:03.924726 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f660a6/simulator-w0-f660a6-kserve-779fbfd878-4wgwt" Sep 12 01:59:04.200691 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:59:04.200655 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18093353_67b9_4a8b_9b50_b07ff688f925.slice/crio-20f60701034ac0d9ca9396071acbc2adbefe819fff50b2a217e98bb0399254a1 WatchSource:0}: Error finding container 20f60701034ac0d9ca9396071acbc2adbefe819fff50b2a217e98bb0399254a1: Status 404 returned error can't find the container with id 20f60701034ac0d9ca9396071acbc2adbefe819fff50b2a217e98bb0399254a1 Sep 12 01:59:04.201607 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:04.201580 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f660a6/simulator-w0-f660a6-kserve-779fbfd878-4wgwt"] Sep 12 01:59:04.896525 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:04.896492 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f660a6/simulator-w0-f660a6-kserve-779fbfd878-4wgwt" event={"ID":"18093353-67b9-4a8b-9b50-b07ff688f925","Type":"ContainerStarted","Data":"20dbe1b095e80f5cbc89d71fee0cd31b5d22d2a70d99a7516832b029c5a6fdb0"} Sep 12 01:59:04.896708 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:04.896539 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f660a6/simulator-w0-f660a6-kserve-779fbfd878-4wgwt" event={"ID":"18093353-67b9-4a8b-9b50-b07ff688f925","Type":"ContainerStarted","Data":"20f60701034ac0d9ca9396071acbc2adbefe819fff50b2a217e98bb0399254a1"} Sep 12 01:59:04.913328 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:04.913273 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f660a6/simulator-w0-f660a6-kserve-779fbfd878-4wgwt" podStartSLOduration=1.9132544550000001 podStartE2EDuration="1.913254455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:59:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:59:04.912369334 +0000 UTC m=+847.704363152" watchObservedRunningTime="2026-09-12 01:59:04.913254455 +0000 UTC m=+847.705248263" Sep 12 01:59:12.832476 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:12.832422 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 01:59:13.925131 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:13.925093 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f660a6/simulator-w0-f660a6-kserve-779fbfd878-4wgwt" Sep 12 01:59:13.925622 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:13.925174 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f660a6/simulator-w0-f660a6-kserve-779fbfd878-4wgwt" Sep 12 01:59:13.938109 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:13.938081 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f660a6/simulator-w0-f660a6-kserve-779fbfd878-4wgwt" Sep 12 01:59:13.957353 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:13.957314 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f660a6/simulator-w0-f660a6-kserve-779fbfd878-4wgwt" Sep 12 01:59:16.860217 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:16.860178 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:59:17.641439 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:17.641402 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:59:18.273880 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:18.273835 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:59:18.931962 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:18.931930 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:59:19.750843 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:19.750804 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:59:20.355264 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:20.355233 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:59:21.179913 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:21.179874 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:59:21.888776 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:21.888744 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:59:22.665800 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:22.665766 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:59:23.438213 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:23.438176 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:59:24.065374 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:24.065334 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:59:24.861698 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:24.861668 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:59:25.511014 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:25.510975 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:59:26.248120 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:26.248084 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:59:32.435388 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:32.435340 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-899a9ad9-666bbd9985-99dz9"] Sep 12 01:59:32.436163 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:32.436120 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-899a9ad9-666bbd9985-99dz9" podUID="a5f01764-9f05-4ed2-8a14-9100827327fa" containerName="maas-api" containerID="cri-o://0d9723d96e6f69e16d87176eec617b8c98a33368532cad416fd3797b027da874" gracePeriod=30 Sep 12 01:59:32.450651 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:32.450618 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:59:32.516849 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:32.516819 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:59:32.580664 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:32.580625 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:59:32.642280 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:32.642245 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:59:32.694725 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:32.694661 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-899a9ad9-666bbd9985-99dz9" Sep 12 01:59:32.797018 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:32.796983 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:59:32.828779 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:32.828750 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5f01764-9f05-4ed2-8a14-9100827327fa-metrics-certs\") pod \"a5f01764-9f05-4ed2-8a14-9100827327fa\" (UID: \"a5f01764-9f05-4ed2-8a14-9100827327fa\") " Sep 12 01:59:32.828960 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:32.828826 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-48vmm\" (UniqueName: \"kubernetes.io/projected/a5f01764-9f05-4ed2-8a14-9100827327fa-kube-api-access-48vmm\") pod \"a5f01764-9f05-4ed2-8a14-9100827327fa\" (UID: \"a5f01764-9f05-4ed2-8a14-9100827327fa\") " Sep 12 01:59:32.828960 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:32.828855 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5f01764-9f05-4ed2-8a14-9100827327fa-openshift-service-ca\") pod \"a5f01764-9f05-4ed2-8a14-9100827327fa\" (UID: \"a5f01764-9f05-4ed2-8a14-9100827327fa\") " Sep 12 01:59:32.829037 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:32.828967 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5f01764-9f05-4ed2-8a14-9100827327fa-maas-api-tls\") pod \"a5f01764-9f05-4ed2-8a14-9100827327fa\" (UID: \"a5f01764-9f05-4ed2-8a14-9100827327fa\") " Sep 12 01:59:32.829314 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:32.829282 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5f01764-9f05-4ed2-8a14-9100827327fa-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a5f01764-9f05-4ed2-8a14-9100827327fa" (UID: "a5f01764-9f05-4ed2-8a14-9100827327fa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 12 01:59:32.829758 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:32.829735 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5f01764-9f05-4ed2-8a14-9100827327fa-openshift-service-ca\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 01:59:32.831559 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:32.831535 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5f01764-9f05-4ed2-8a14-9100827327fa-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a5f01764-9f05-4ed2-8a14-9100827327fa" (UID: "a5f01764-9f05-4ed2-8a14-9100827327fa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 12 01:59:32.831901 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:32.831869 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5f01764-9f05-4ed2-8a14-9100827327fa-kube-api-access-48vmm" (OuterVolumeSpecName: "kube-api-access-48vmm") pod "a5f01764-9f05-4ed2-8a14-9100827327fa" (UID: "a5f01764-9f05-4ed2-8a14-9100827327fa"). InnerVolumeSpecName "kube-api-access-48vmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 12 01:59:32.832079 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:32.832058 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5f01764-9f05-4ed2-8a14-9100827327fa-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a5f01764-9f05-4ed2-8a14-9100827327fa" (UID: "a5f01764-9f05-4ed2-8a14-9100827327fa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 12 01:59:32.930464 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:32.930427 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5f01764-9f05-4ed2-8a14-9100827327fa-maas-api-tls\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 01:59:32.930633 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:32.930488 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5f01764-9f05-4ed2-8a14-9100827327fa-metrics-certs\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 01:59:32.930633 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:32.930497 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48vmm\" (UniqueName: \"kubernetes.io/projected/a5f01764-9f05-4ed2-8a14-9100827327fa-kube-api-access-48vmm\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 01:59:33.048272 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:33.048164 2579 generic.go:356] "Generic (PLEG): container finished" podID="a5f01764-9f05-4ed2-8a14-9100827327fa" containerID="0d9723d96e6f69e16d87176eec617b8c98a33368532cad416fd3797b027da874" exitCode=0 Sep 12 01:59:33.048468 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:33.048357 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-899a9ad9-666bbd9985-99dz9" event={"ID":"a5f01764-9f05-4ed2-8a14-9100827327fa","Type":"ContainerDied","Data":"0d9723d96e6f69e16d87176eec617b8c98a33368532cad416fd3797b027da874"} Sep 12 01:59:33.048536 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:33.048466 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-899a9ad9-666bbd9985-99dz9" event={"ID":"a5f01764-9f05-4ed2-8a14-9100827327fa","Type":"ContainerDied","Data":"b2d28af4419b65bb4158de1eb405d4dd73bbb3d8e36f54931382f3b540c1f25d"} Sep 12 01:59:33.048536 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:33.048508 2579 scope.go:117] "RemoveContainer" containerID="0d9723d96e6f69e16d87176eec617b8c98a33368532cad416fd3797b027da874" Sep 12 01:59:33.048734 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:33.048715 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-899a9ad9-666bbd9985-99dz9" Sep 12 01:59:33.059351 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:33.059327 2579 scope.go:117] "RemoveContainer" containerID="0d9723d96e6f69e16d87176eec617b8c98a33368532cad416fd3797b027da874" Sep 12 01:59:33.059681 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:59:33.059659 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d9723d96e6f69e16d87176eec617b8c98a33368532cad416fd3797b027da874\": container with ID starting with 0d9723d96e6f69e16d87176eec617b8c98a33368532cad416fd3797b027da874 not found: ID does not exist" containerID="0d9723d96e6f69e16d87176eec617b8c98a33368532cad416fd3797b027da874" Sep 12 01:59:33.059756 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:33.059690 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d9723d96e6f69e16d87176eec617b8c98a33368532cad416fd3797b027da874"} err="failed to get container status \"0d9723d96e6f69e16d87176eec617b8c98a33368532cad416fd3797b027da874\": rpc error: code = NotFound desc = could not find container \"0d9723d96e6f69e16d87176eec617b8c98a33368532cad416fd3797b027da874\": container with ID starting with 0d9723d96e6f69e16d87176eec617b8c98a33368532cad416fd3797b027da874 not found: ID does not exist" Sep 12 01:59:33.071367 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:33.071339 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-899a9ad9-666bbd9985-99dz9"] Sep 12 01:59:33.072852 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:33.072831 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-899a9ad9-666bbd9985-99dz9"] Sep 12 01:59:33.080494 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:33.080439 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:59:33.269731 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:33.269698 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:59:33.594524 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:33.594487 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:59:33.727408 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:33.727370 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:59:33.727940 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:33.727919 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5f01764-9f05-4ed2-8a14-9100827327fa" path="/var/lib/kubelet/pods/a5f01764-9f05-4ed2-8a14-9100827327fa/volumes" Sep 12 01:59:33.857616 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:33.857505 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:59:34.736570 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:34.736531 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:59:35.634254 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:35.634221 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:59:36.256119 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:36.256085 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:59:36.354831 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:36.354797 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:59:37.264201 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:37.264166 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:59:37.369553 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:37.369515 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:59:38.024420 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:38.024389 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-29d2f735-65874bfddd-q949x"] Sep 12 01:59:38.024861 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:38.024842 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5f01764-9f05-4ed2-8a14-9100827327fa" containerName="maas-api" Sep 12 01:59:38.024861 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:38.024857 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5f01764-9f05-4ed2-8a14-9100827327fa" containerName="maas-api" Sep 12 01:59:38.025042 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:38.024926 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5f01764-9f05-4ed2-8a14-9100827327fa" containerName="maas-api" Sep 12 01:59:38.029544 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:38.029519 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-29d2f735-65874bfddd-q949x" Sep 12 01:59:38.031889 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:38.031868 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-29d2f735-serving-cert\"" Sep 12 01:59:38.040828 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:38.040803 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-29d2f735-65874bfddd-q949x"] Sep 12 01:59:38.055175 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:38.055142 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:59:38.154638 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:38.154605 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:59:38.184040 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:38.184004 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29e129dd-b3e6-4622-a808-586709e5a871-openshift-service-ca\") pod \"maas-api-e2e-ait-29d2f735-65874bfddd-q949x\" (UID: \"29e129dd-b3e6-4622-a808-586709e5a871\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-29d2f735-65874bfddd-q949x" Sep 12 01:59:38.184040 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:38.184042 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5txh\" (UniqueName: \"kubernetes.io/projected/29e129dd-b3e6-4622-a808-586709e5a871-kube-api-access-z5txh\") pod \"maas-api-e2e-ait-29d2f735-65874bfddd-q949x\" (UID: \"29e129dd-b3e6-4622-a808-586709e5a871\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-29d2f735-65874bfddd-q949x" Sep 12 01:59:38.184306 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:38.184164 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29e129dd-b3e6-4622-a808-586709e5a871-maas-api-tls\") pod \"maas-api-e2e-ait-29d2f735-65874bfddd-q949x\" (UID: \"29e129dd-b3e6-4622-a808-586709e5a871\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-29d2f735-65874bfddd-q949x" Sep 12 01:59:38.184306 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:38.184199 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29e129dd-b3e6-4622-a808-586709e5a871-metrics-certs\") pod \"maas-api-e2e-ait-29d2f735-65874bfddd-q949x\" (UID: \"29e129dd-b3e6-4622-a808-586709e5a871\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-29d2f735-65874bfddd-q949x" Sep 12 01:59:38.285026 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:38.284942 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29e129dd-b3e6-4622-a808-586709e5a871-openshift-service-ca\") pod \"maas-api-e2e-ait-29d2f735-65874bfddd-q949x\" (UID: \"29e129dd-b3e6-4622-a808-586709e5a871\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-29d2f735-65874bfddd-q949x" Sep 12 01:59:38.285026 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:38.284981 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5txh\" (UniqueName: \"kubernetes.io/projected/29e129dd-b3e6-4622-a808-586709e5a871-kube-api-access-z5txh\") pod \"maas-api-e2e-ait-29d2f735-65874bfddd-q949x\" (UID: \"29e129dd-b3e6-4622-a808-586709e5a871\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-29d2f735-65874bfddd-q949x" Sep 12 01:59:38.285490 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:38.285077 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29e129dd-b3e6-4622-a808-586709e5a871-maas-api-tls\") pod \"maas-api-e2e-ait-29d2f735-65874bfddd-q949x\" (UID: \"29e129dd-b3e6-4622-a808-586709e5a871\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-29d2f735-65874bfddd-q949x" Sep 12 01:59:38.285490 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:38.285119 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29e129dd-b3e6-4622-a808-586709e5a871-metrics-certs\") pod \"maas-api-e2e-ait-29d2f735-65874bfddd-q949x\" (UID: \"29e129dd-b3e6-4622-a808-586709e5a871\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-29d2f735-65874bfddd-q949x" Sep 12 01:59:38.285490 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:59:38.285223 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-29d2f735-serving-cert: secret "maas-api-e2e-ait-29d2f735-serving-cert" not found Sep 12 01:59:38.285490 ip-10-0-133-79 kubenswrapper[2579]: E0912 01:59:38.285299 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29e129dd-b3e6-4622-a808-586709e5a871-maas-api-tls podName:29e129dd-b3e6-4622-a808-586709e5a871 nodeName:}" failed. No retries permitted until 2026-09-12 01:59:38.785277683 +0000 UTC m=+881.577271461 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/29e129dd-b3e6-4622-a808-586709e5a871-maas-api-tls") pod "maas-api-e2e-ait-29d2f735-65874bfddd-q949x" (UID: "29e129dd-b3e6-4622-a808-586709e5a871") : secret "maas-api-e2e-ait-29d2f735-serving-cert" not found Sep 12 01:59:38.285705 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:38.285684 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29e129dd-b3e6-4622-a808-586709e5a871-openshift-service-ca\") pod \"maas-api-e2e-ait-29d2f735-65874bfddd-q949x\" (UID: \"29e129dd-b3e6-4622-a808-586709e5a871\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-29d2f735-65874bfddd-q949x" Sep 12 01:59:38.287666 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:38.287647 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29e129dd-b3e6-4622-a808-586709e5a871-metrics-certs\") pod \"maas-api-e2e-ait-29d2f735-65874bfddd-q949x\" (UID: \"29e129dd-b3e6-4622-a808-586709e5a871\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-29d2f735-65874bfddd-q949x" Sep 12 01:59:38.297395 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:38.297372 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5txh\" (UniqueName: \"kubernetes.io/projected/29e129dd-b3e6-4622-a808-586709e5a871-kube-api-access-z5txh\") pod \"maas-api-e2e-ait-29d2f735-65874bfddd-q949x\" (UID: \"29e129dd-b3e6-4622-a808-586709e5a871\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-29d2f735-65874bfddd-q949x" Sep 12 01:59:38.426424 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:38.426390 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:59:38.583181 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:38.583068 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:59:38.681695 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:38.680678 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-29d2f735-857db5894-tb86z"] Sep 12 01:59:38.681695 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:38.681043 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:59:38.686810 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:38.686782 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-29d2f735-857db5894-tb86z" Sep 12 01:59:38.689040 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:38.689016 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-29d2f735\"" Sep 12 01:59:38.689862 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:38.689837 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-29d2f735-857db5894-tb86z"] Sep 12 01:59:38.789075 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:38.789043 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29e129dd-b3e6-4622-a808-586709e5a871-maas-api-tls\") pod \"maas-api-e2e-ait-29d2f735-65874bfddd-q949x\" (UID: \"29e129dd-b3e6-4622-a808-586709e5a871\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-29d2f735-65874bfddd-q949x" Sep 12 01:59:38.789252 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:38.789125 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwrb5\" (UniqueName: \"kubernetes.io/projected/fcf16f7c-84e7-4fa2-b611-a13d6b38fad4-kube-api-access-dwrb5\") pod \"payload-pre-processing-e2e-ait-29d2f735-857db5894-tb86z\" (UID: \"fcf16f7c-84e7-4fa2-b611-a13d6b38fad4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-29d2f735-857db5894-tb86z" Sep 12 01:59:38.789252 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:38.789184 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcf16f7c-84e7-4fa2-b611-a13d6b38fad4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-29d2f735-857db5894-tb86z\" (UID: \"fcf16f7c-84e7-4fa2-b611-a13d6b38fad4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-29d2f735-857db5894-tb86z" Sep 12 01:59:38.791770 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:38.791742 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29e129dd-b3e6-4622-a808-586709e5a871-maas-api-tls\") pod \"maas-api-e2e-ait-29d2f735-65874bfddd-q949x\" (UID: \"29e129dd-b3e6-4622-a808-586709e5a871\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-29d2f735-65874bfddd-q949x" Sep 12 01:59:38.890296 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:38.890255 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwrb5\" (UniqueName: \"kubernetes.io/projected/fcf16f7c-84e7-4fa2-b611-a13d6b38fad4-kube-api-access-dwrb5\") pod \"payload-pre-processing-e2e-ait-29d2f735-857db5894-tb86z\" (UID: \"fcf16f7c-84e7-4fa2-b611-a13d6b38fad4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-29d2f735-857db5894-tb86z" Sep 12 01:59:38.890530 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:38.890336 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcf16f7c-84e7-4fa2-b611-a13d6b38fad4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-29d2f735-857db5894-tb86z\" (UID: \"fcf16f7c-84e7-4fa2-b611-a13d6b38fad4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-29d2f735-857db5894-tb86z" Sep 12 01:59:38.891004 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:38.890984 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcf16f7c-84e7-4fa2-b611-a13d6b38fad4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-29d2f735-857db5894-tb86z\" (UID: \"fcf16f7c-84e7-4fa2-b611-a13d6b38fad4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-29d2f735-857db5894-tb86z" Sep 12 01:59:38.898510 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:38.898481 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwrb5\" (UniqueName: \"kubernetes.io/projected/fcf16f7c-84e7-4fa2-b611-a13d6b38fad4-kube-api-access-dwrb5\") pod \"payload-pre-processing-e2e-ait-29d2f735-857db5894-tb86z\" (UID: \"fcf16f7c-84e7-4fa2-b611-a13d6b38fad4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-29d2f735-857db5894-tb86z" Sep 12 01:59:38.943094 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:38.943051 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-29d2f735-65874bfddd-q949x" Sep 12 01:59:38.957977 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:38.957928 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:59:38.998924 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:38.998591 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-29d2f735-857db5894-tb86z" Sep 12 01:59:39.220562 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:39.220538 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-29d2f735-65874bfddd-q949x"] Sep 12 01:59:39.222637 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:59:39.222613 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29e129dd_b3e6_4622_a808_586709e5a871.slice/crio-ca15f19cacfba4116872287af4bfbf431f9883752cfc465eb17bac367b5e23e7 WatchSource:0}: Error finding container ca15f19cacfba4116872287af4bfbf431f9883752cfc465eb17bac367b5e23e7: Status 404 returned error can't find the container with id ca15f19cacfba4116872287af4bfbf431f9883752cfc465eb17bac367b5e23e7 Sep 12 01:59:39.276037 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:39.276013 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-29d2f735-857db5894-tb86z"] Sep 12 01:59:39.277782 ip-10-0-133-79 kubenswrapper[2579]: W0912 01:59:39.277744 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcf16f7c_84e7_4fa2_b611_a13d6b38fad4.slice/crio-661aae393ea659ea23c601a9f01b9984554e78ed1836f181b7fc884ff0e43cc2 WatchSource:0}: Error finding container 661aae393ea659ea23c601a9f01b9984554e78ed1836f181b7fc884ff0e43cc2: Status 404 returned error can't find the container with id 661aae393ea659ea23c601a9f01b9984554e78ed1836f181b7fc884ff0e43cc2 Sep 12 01:59:39.330036 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:39.330001 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 01:59:39.872690 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:39.872660 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:59:40.089433 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:40.089389 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-29d2f735-65874bfddd-q949x" event={"ID":"29e129dd-b3e6-4622-a808-586709e5a871","Type":"ContainerStarted","Data":"8f5366e3ec660a0d6ae789a066893843860eb104121d38cc87fa7da52456cafc"} Sep 12 01:59:40.089629 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:40.089601 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-29d2f735-65874bfddd-q949x" Sep 12 01:59:40.089700 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:40.089647 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-29d2f735-65874bfddd-q949x" event={"ID":"29e129dd-b3e6-4622-a808-586709e5a871","Type":"ContainerStarted","Data":"ca15f19cacfba4116872287af4bfbf431f9883752cfc465eb17bac367b5e23e7"} Sep 12 01:59:40.091024 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:40.091003 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-29d2f735-857db5894-tb86z" event={"ID":"fcf16f7c-84e7-4fa2-b611-a13d6b38fad4","Type":"ContainerStarted","Data":"c41ba786e7188ffc38a592b84f336257cb06571cd4f833107a23d4e905b0b08b"} Sep 12 01:59:40.091128 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:40.091038 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-29d2f735-857db5894-tb86z" event={"ID":"fcf16f7c-84e7-4fa2-b611-a13d6b38fad4","Type":"ContainerStarted","Data":"661aae393ea659ea23c601a9f01b9984554e78ed1836f181b7fc884ff0e43cc2"} Sep 12 01:59:40.091192 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:40.091141 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-29d2f735-857db5894-tb86z" Sep 12 01:59:40.107711 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:40.107651 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-29d2f735-65874bfddd-q949x" podStartSLOduration=1.766113903 podStartE2EDuration="2.107633324s" totalImagesPullingTime="341.519421ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:59:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:59:40.106227386 +0000 UTC m=+882.898221180" watchObservedRunningTime="2026-09-12 01:59:40.107633324 +0000 UTC m=+882.899627121" Sep 12 01:59:40.123517 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:40.123381 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-29d2f735-857db5894-tb86z" podStartSLOduration=2.123366298 podStartE2EDuration="2.123366298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:59:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:59:40.121121783 +0000 UTC m=+882.913115581" watchObservedRunningTime="2026-09-12 01:59:40.123366298 +0000 UTC m=+882.915360093" Sep 12 01:59:40.549983 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:40.549894 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:59:41.272178 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:41.272136 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:59:41.925372 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:41.925337 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 01:59:46.101560 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:46.101531 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-29d2f735-65874bfddd-q949x" Sep 12 01:59:51.098257 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:51.098164 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-29d2f735-857db5894-tb86z" Sep 12 01:59:58.725123 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:58.725086 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 01:59:59.948940 ip-10-0-133-79 kubenswrapper[2579]: I0912 01:59:59.948900 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:00.155484 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:00.155431 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819640-868qh"] Sep 12 02:00:00.160525 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:00.160499 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819640-868qh" Sep 12 02:00:00.162801 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:00.162780 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7x88s\"" Sep 12 02:00:00.163318 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:00.163296 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f660a6-29819640-pfmd7"] Sep 12 02:00:00.167332 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:00.167308 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f660a6-29819640-pfmd7" Sep 12 02:00:00.171288 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:00.170917 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819640-868qh"] Sep 12 02:00:00.177101 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:00.177079 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819640-868qh" Sep 12 02:00:00.178043 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:00.178021 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f660a6-29819640-pfmd7"] Sep 12 02:00:00.185846 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:00.185827 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f660a6-29819640-pfmd7" Sep 12 02:00:00.460847 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:00.460812 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819640-868qh"] Sep 12 02:00:00.462999 ip-10-0-133-79 kubenswrapper[2579]: W0912 02:00:00.462963 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa80bfbd_b345_47d0_8f68_9c680229276a.slice/crio-92f5fa4ce3ae7d46a3db2d57d8f7b0c0ab4b2808cde084daf0eec27a31b474b3 WatchSource:0}: Error finding container 92f5fa4ce3ae7d46a3db2d57d8f7b0c0ab4b2808cde084daf0eec27a31b474b3: Status 404 returned error can't find the container with id 92f5fa4ce3ae7d46a3db2d57d8f7b0c0ab4b2808cde084daf0eec27a31b474b3 Sep 12 02:00:00.476238 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:00.476210 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f660a6-29819640-pfmd7"] Sep 12 02:00:00.477298 ip-10-0-133-79 kubenswrapper[2579]: W0912 02:00:00.477265 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc544667_2a18_453f_8e5d_9456f346a6da.slice/crio-6a7dec767096e57bd9df3c418054f031d856cac76d49c3b9ebe7b5d704b05ace WatchSource:0}: Error finding container 6a7dec767096e57bd9df3c418054f031d856cac76d49c3b9ebe7b5d704b05ace: Status 404 returned error can't find the container with id 6a7dec767096e57bd9df3c418054f031d856cac76d49c3b9ebe7b5d704b05ace Sep 12 02:00:00.716003 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:00.715917 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:01.203514 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:01.203480 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f660a6-29819640-pfmd7" event={"ID":"fc544667-2a18-453f-8e5d-9456f346a6da","Type":"ContainerStarted","Data":"6a7dec767096e57bd9df3c418054f031d856cac76d49c3b9ebe7b5d704b05ace"} Sep 12 02:00:01.204746 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:01.204724 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819640-868qh" event={"ID":"fa80bfbd-b345-47d0-8f68-9c680229276a","Type":"ContainerStarted","Data":"92f5fa4ce3ae7d46a3db2d57d8f7b0c0ab4b2808cde084daf0eec27a31b474b3"} Sep 12 02:00:01.367998 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:01.367967 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:02.060028 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:02.059958 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:03.118029 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:03.117998 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:03.639169 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:03.639125 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:04.224403 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:04.224372 2579 generic.go:356] "Generic (PLEG): container finished" podID="fc544667-2a18-453f-8e5d-9456f346a6da" containerID="84d3ed5fec90d2131378fcaba499e7eb57219bfadaac578e979b89813a1f7ef4" exitCode=0 Sep 12 02:00:04.224869 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:04.224482 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f660a6-29819640-pfmd7" event={"ID":"fc544667-2a18-453f-8e5d-9456f346a6da","Type":"ContainerDied","Data":"84d3ed5fec90d2131378fcaba499e7eb57219bfadaac578e979b89813a1f7ef4"} Sep 12 02:00:04.226225 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:04.226191 2579 generic.go:356] "Generic (PLEG): container finished" podID="fa80bfbd-b345-47d0-8f68-9c680229276a" containerID="0c5505fae71071a513246da070f98f8dfb3fd63e985a77b769cd3dddd19e9bcd" exitCode=0 Sep 12 02:00:04.226336 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:04.226276 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819640-868qh" event={"ID":"fa80bfbd-b345-47d0-8f68-9c680229276a","Type":"ContainerDied","Data":"0c5505fae71071a513246da070f98f8dfb3fd63e985a77b769cd3dddd19e9bcd"} Sep 12 02:00:04.404996 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:04.404958 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:05.063191 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:05.063152 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:05.387438 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:05.387414 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819640-868qh" Sep 12 02:00:05.390492 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:05.390474 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f660a6-29819640-pfmd7" Sep 12 02:00:05.716923 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:05.716889 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:06.238206 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:06.238174 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f660a6-29819640-pfmd7" Sep 12 02:00:06.238206 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:06.238188 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f660a6-29819640-pfmd7" event={"ID":"fc544667-2a18-453f-8e5d-9456f346a6da","Type":"ContainerDied","Data":"6a7dec767096e57bd9df3c418054f031d856cac76d49c3b9ebe7b5d704b05ace"} Sep 12 02:00:06.238445 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:06.238235 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a7dec767096e57bd9df3c418054f031d856cac76d49c3b9ebe7b5d704b05ace" Sep 12 02:00:06.239735 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:06.239717 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819640-868qh" Sep 12 02:00:06.239841 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:06.239768 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819640-868qh" event={"ID":"fa80bfbd-b345-47d0-8f68-9c680229276a","Type":"ContainerDied","Data":"92f5fa4ce3ae7d46a3db2d57d8f7b0c0ab4b2808cde084daf0eec27a31b474b3"} Sep 12 02:00:06.239841 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:06.239803 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92f5fa4ce3ae7d46a3db2d57d8f7b0c0ab4b2808cde084daf0eec27a31b474b3" Sep 12 02:00:06.455925 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:06.455888 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:07.178672 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:07.178630 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:07.704283 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:07.704249 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-29d2f735-65874bfddd-q949x"] Sep 12 02:00:07.704767 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:07.704563 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-29d2f735-65874bfddd-q949x" podUID="29e129dd-b3e6-4622-a808-586709e5a871" containerName="maas-api" containerID="cri-o://8f5366e3ec660a0d6ae789a066893843860eb104121d38cc87fa7da52456cafc" gracePeriod=30 Sep 12 02:00:07.736424 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:07.736396 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:07.780303 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:07.780271 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-29d2f735-857db5894-tb86z"] Sep 12 02:00:07.780566 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:07.780537 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-29d2f735-857db5894-tb86z" podUID="fcf16f7c-84e7-4fa2-b611-a13d6b38fad4" containerName="payload-pre-processing" containerID="cri-o://c41ba786e7188ffc38a592b84f336257cb06571cd4f833107a23d4e905b0b08b" gracePeriod=30 Sep 12 02:00:07.836328 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:07.836296 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:07.931398 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:07.931364 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:07.989952 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:07.989870 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:08.030202 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:08.030169 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-29d2f735-65874bfddd-q949x" Sep 12 02:00:08.063971 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:08.063948 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-29d2f735-857db5894-tb86z" Sep 12 02:00:08.090481 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:08.090435 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29e129dd-b3e6-4622-a808-586709e5a871-metrics-certs\") pod \"29e129dd-b3e6-4622-a808-586709e5a871\" (UID: \"29e129dd-b3e6-4622-a808-586709e5a871\") " Sep 12 02:00:08.090673 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:08.090520 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29e129dd-b3e6-4622-a808-586709e5a871-maas-api-tls\") pod \"29e129dd-b3e6-4622-a808-586709e5a871\" (UID: \"29e129dd-b3e6-4622-a808-586709e5a871\") " Sep 12 02:00:08.090673 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:08.090563 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29e129dd-b3e6-4622-a808-586709e5a871-openshift-service-ca\") pod \"29e129dd-b3e6-4622-a808-586709e5a871\" (UID: \"29e129dd-b3e6-4622-a808-586709e5a871\") " Sep 12 02:00:08.090673 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:08.090656 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z5txh\" (UniqueName: \"kubernetes.io/projected/29e129dd-b3e6-4622-a808-586709e5a871-kube-api-access-z5txh\") pod \"29e129dd-b3e6-4622-a808-586709e5a871\" (UID: \"29e129dd-b3e6-4622-a808-586709e5a871\") " Sep 12 02:00:08.090900 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:08.090874 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29e129dd-b3e6-4622-a808-586709e5a871-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "29e129dd-b3e6-4622-a808-586709e5a871" (UID: "29e129dd-b3e6-4622-a808-586709e5a871"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 12 02:00:08.091093 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:08.091074 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29e129dd-b3e6-4622-a808-586709e5a871-openshift-service-ca\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:00:08.093218 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:08.093178 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29e129dd-b3e6-4622-a808-586709e5a871-kube-api-access-z5txh" (OuterVolumeSpecName: "kube-api-access-z5txh") pod "29e129dd-b3e6-4622-a808-586709e5a871" (UID: "29e129dd-b3e6-4622-a808-586709e5a871"). InnerVolumeSpecName "kube-api-access-z5txh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 12 02:00:08.093373 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:08.093352 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29e129dd-b3e6-4622-a808-586709e5a871-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "29e129dd-b3e6-4622-a808-586709e5a871" (UID: "29e129dd-b3e6-4622-a808-586709e5a871"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 12 02:00:08.093442 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:08.093373 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29e129dd-b3e6-4622-a808-586709e5a871-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "29e129dd-b3e6-4622-a808-586709e5a871" (UID: "29e129dd-b3e6-4622-a808-586709e5a871"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 12 02:00:08.192003 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:08.191966 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dwrb5\" (UniqueName: \"kubernetes.io/projected/fcf16f7c-84e7-4fa2-b611-a13d6b38fad4-kube-api-access-dwrb5\") pod \"fcf16f7c-84e7-4fa2-b611-a13d6b38fad4\" (UID: \"fcf16f7c-84e7-4fa2-b611-a13d6b38fad4\") " Sep 12 02:00:08.192182 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:08.192027 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcf16f7c-84e7-4fa2-b611-a13d6b38fad4-plugins-config-volume\") pod \"fcf16f7c-84e7-4fa2-b611-a13d6b38fad4\" (UID: \"fcf16f7c-84e7-4fa2-b611-a13d6b38fad4\") " Sep 12 02:00:08.192320 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:08.192306 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5txh\" (UniqueName: \"kubernetes.io/projected/29e129dd-b3e6-4622-a808-586709e5a871-kube-api-access-z5txh\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:00:08.192358 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:08.192326 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29e129dd-b3e6-4622-a808-586709e5a871-metrics-certs\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:00:08.192358 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:08.192340 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29e129dd-b3e6-4622-a808-586709e5a871-maas-api-tls\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:00:08.192418 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:08.192388 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcf16f7c-84e7-4fa2-b611-a13d6b38fad4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fcf16f7c-84e7-4fa2-b611-a13d6b38fad4" (UID: "fcf16f7c-84e7-4fa2-b611-a13d6b38fad4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 12 02:00:08.194426 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:08.194405 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcf16f7c-84e7-4fa2-b611-a13d6b38fad4-kube-api-access-dwrb5" (OuterVolumeSpecName: "kube-api-access-dwrb5") pod "fcf16f7c-84e7-4fa2-b611-a13d6b38fad4" (UID: "fcf16f7c-84e7-4fa2-b611-a13d6b38fad4"). InnerVolumeSpecName "kube-api-access-dwrb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 12 02:00:08.253687 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:08.253606 2579 generic.go:356] "Generic (PLEG): container finished" podID="fcf16f7c-84e7-4fa2-b611-a13d6b38fad4" containerID="c41ba786e7188ffc38a592b84f336257cb06571cd4f833107a23d4e905b0b08b" exitCode=0 Sep 12 02:00:08.253687 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:08.253654 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-29d2f735-857db5894-tb86z" Sep 12 02:00:08.253887 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:08.253691 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-29d2f735-857db5894-tb86z" event={"ID":"fcf16f7c-84e7-4fa2-b611-a13d6b38fad4","Type":"ContainerDied","Data":"c41ba786e7188ffc38a592b84f336257cb06571cd4f833107a23d4e905b0b08b"} Sep 12 02:00:08.253887 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:08.253744 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-29d2f735-857db5894-tb86z" event={"ID":"fcf16f7c-84e7-4fa2-b611-a13d6b38fad4","Type":"ContainerDied","Data":"661aae393ea659ea23c601a9f01b9984554e78ed1836f181b7fc884ff0e43cc2"} Sep 12 02:00:08.253887 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:08.253775 2579 scope.go:117] "RemoveContainer" containerID="c41ba786e7188ffc38a592b84f336257cb06571cd4f833107a23d4e905b0b08b" Sep 12 02:00:08.255307 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:08.255270 2579 generic.go:356] "Generic (PLEG): container finished" podID="29e129dd-b3e6-4622-a808-586709e5a871" containerID="8f5366e3ec660a0d6ae789a066893843860eb104121d38cc87fa7da52456cafc" exitCode=0 Sep 12 02:00:08.255426 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:08.255332 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-29d2f735-65874bfddd-q949x" Sep 12 02:00:08.255426 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:08.255331 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-29d2f735-65874bfddd-q949x" event={"ID":"29e129dd-b3e6-4622-a808-586709e5a871","Type":"ContainerDied","Data":"8f5366e3ec660a0d6ae789a066893843860eb104121d38cc87fa7da52456cafc"} Sep 12 02:00:08.255528 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:08.255472 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-29d2f735-65874bfddd-q949x" event={"ID":"29e129dd-b3e6-4622-a808-586709e5a871","Type":"ContainerDied","Data":"ca15f19cacfba4116872287af4bfbf431f9883752cfc465eb17bac367b5e23e7"} Sep 12 02:00:08.263282 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:08.263265 2579 scope.go:117] "RemoveContainer" containerID="c41ba786e7188ffc38a592b84f336257cb06571cd4f833107a23d4e905b0b08b" Sep 12 02:00:08.263659 ip-10-0-133-79 kubenswrapper[2579]: E0912 02:00:08.263635 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c41ba786e7188ffc38a592b84f336257cb06571cd4f833107a23d4e905b0b08b\": container with ID starting with c41ba786e7188ffc38a592b84f336257cb06571cd4f833107a23d4e905b0b08b not found: ID does not exist" containerID="c41ba786e7188ffc38a592b84f336257cb06571cd4f833107a23d4e905b0b08b" Sep 12 02:00:08.263749 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:08.263669 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c41ba786e7188ffc38a592b84f336257cb06571cd4f833107a23d4e905b0b08b"} err="failed to get container status \"c41ba786e7188ffc38a592b84f336257cb06571cd4f833107a23d4e905b0b08b\": rpc error: code = NotFound desc = could not find container \"c41ba786e7188ffc38a592b84f336257cb06571cd4f833107a23d4e905b0b08b\": container with ID starting with c41ba786e7188ffc38a592b84f336257cb06571cd4f833107a23d4e905b0b08b not found: ID does not exist" Sep 12 02:00:08.263749 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:08.263685 2579 scope.go:117] "RemoveContainer" containerID="8f5366e3ec660a0d6ae789a066893843860eb104121d38cc87fa7da52456cafc" Sep 12 02:00:08.273296 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:08.273274 2579 scope.go:117] "RemoveContainer" containerID="8f5366e3ec660a0d6ae789a066893843860eb104121d38cc87fa7da52456cafc" Sep 12 02:00:08.273646 ip-10-0-133-79 kubenswrapper[2579]: E0912 02:00:08.273626 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f5366e3ec660a0d6ae789a066893843860eb104121d38cc87fa7da52456cafc\": container with ID starting with 8f5366e3ec660a0d6ae789a066893843860eb104121d38cc87fa7da52456cafc not found: ID does not exist" containerID="8f5366e3ec660a0d6ae789a066893843860eb104121d38cc87fa7da52456cafc" Sep 12 02:00:08.273716 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:08.273654 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f5366e3ec660a0d6ae789a066893843860eb104121d38cc87fa7da52456cafc"} err="failed to get container status \"8f5366e3ec660a0d6ae789a066893843860eb104121d38cc87fa7da52456cafc\": rpc error: code = NotFound desc = could not find container \"8f5366e3ec660a0d6ae789a066893843860eb104121d38cc87fa7da52456cafc\": container with ID starting with 8f5366e3ec660a0d6ae789a066893843860eb104121d38cc87fa7da52456cafc not found: ID does not exist" Sep 12 02:00:08.280796 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:08.280773 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-29d2f735-65874bfddd-q949x"] Sep 12 02:00:08.287680 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:08.287656 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-29d2f735-65874bfddd-q949x"] Sep 12 02:00:08.293105 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:08.293082 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcf16f7c-84e7-4fa2-b611-a13d6b38fad4-plugins-config-volume\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:00:08.293253 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:08.293107 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwrb5\" (UniqueName: \"kubernetes.io/projected/fcf16f7c-84e7-4fa2-b611-a13d6b38fad4-kube-api-access-dwrb5\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:00:08.300129 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:08.300101 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-29d2f735-857db5894-tb86z"] Sep 12 02:00:08.302113 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:08.302091 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-29d2f735-857db5894-tb86z"] Sep 12 02:00:08.367376 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:08.367338 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:09.032954 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:09.032857 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:09.200939 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:09.200903 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:09.727675 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:09.727638 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29e129dd-b3e6-4622-a808-586709e5a871" path="/var/lib/kubelet/pods/29e129dd-b3e6-4622-a808-586709e5a871/volumes" Sep 12 02:00:09.728247 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:09.728218 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcf16f7c-84e7-4fa2-b611-a13d6b38fad4" path="/var/lib/kubelet/pods/fcf16f7c-84e7-4fa2-b611-a13d6b38fad4/volumes" Sep 12 02:00:09.728497 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:09.728470 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:09.948996 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:09.948949 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:10.053598 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:10.053502 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:10.144987 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:10.144953 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:10.250297 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:10.250262 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:10.828268 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:10.828230 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:11.773744 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:11.773713 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:13.117053 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:13.117019 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:13.490271 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:13.490239 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:14.356815 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:14.356785 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:14.988412 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:14.988377 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:15.034633 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:15.034602 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:00:15.853075 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:15.853027 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:16.652040 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:16.652002 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:17.464587 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:17.464550 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e557ff8f-d9567988c-pfzgn"] Sep 12 02:00:17.465117 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:17.465093 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcf16f7c-84e7-4fa2-b611-a13d6b38fad4" containerName="payload-pre-processing" Sep 12 02:00:17.465117 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:17.465116 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcf16f7c-84e7-4fa2-b611-a13d6b38fad4" containerName="payload-pre-processing" Sep 12 02:00:17.465284 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:17.465142 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa80bfbd-b345-47d0-8f68-9c680229276a" containerName="cleanup" Sep 12 02:00:17.465284 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:17.465151 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa80bfbd-b345-47d0-8f68-9c680229276a" containerName="cleanup" Sep 12 02:00:17.465284 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:17.465161 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc544667-2a18-453f-8e5d-9456f346a6da" containerName="cleanup" Sep 12 02:00:17.465284 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:17.465166 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc544667-2a18-453f-8e5d-9456f346a6da" containerName="cleanup" Sep 12 02:00:17.465284 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:17.465176 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29e129dd-b3e6-4622-a808-586709e5a871" containerName="maas-api" Sep 12 02:00:17.465284 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:17.465181 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="29e129dd-b3e6-4622-a808-586709e5a871" containerName="maas-api" Sep 12 02:00:17.465598 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:17.465294 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa80bfbd-b345-47d0-8f68-9c680229276a" containerName="cleanup" Sep 12 02:00:17.465598 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:17.465312 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="29e129dd-b3e6-4622-a808-586709e5a871" containerName="maas-api" Sep 12 02:00:17.465598 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:17.465321 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc544667-2a18-453f-8e5d-9456f346a6da" containerName="cleanup" Sep 12 02:00:17.465598 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:17.465330 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcf16f7c-84e7-4fa2-b611-a13d6b38fad4" containerName="payload-pre-processing" Sep 12 02:00:17.469254 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:17.469237 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e557ff8f-d9567988c-pfzgn" Sep 12 02:00:17.472404 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:17.472378 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-e557ff8f-serving-cert\"" Sep 12 02:00:17.481421 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:17.481399 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e557ff8f-d9567988c-pfzgn"] Sep 12 02:00:17.499007 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:17.498973 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:17.597028 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:17.596991 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bb564b0-14dd-41e4-9cb5-419abfd099d1-openshift-service-ca\") pod \"maas-api-e2e-derive-e557ff8f-d9567988c-pfzgn\" (UID: \"7bb564b0-14dd-41e4-9cb5-419abfd099d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e557ff8f-d9567988c-pfzgn" Sep 12 02:00:17.597224 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:17.597069 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bb564b0-14dd-41e4-9cb5-419abfd099d1-maas-api-tls\") pod \"maas-api-e2e-derive-e557ff8f-d9567988c-pfzgn\" (UID: \"7bb564b0-14dd-41e4-9cb5-419abfd099d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e557ff8f-d9567988c-pfzgn" Sep 12 02:00:17.597224 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:17.597129 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bb564b0-14dd-41e4-9cb5-419abfd099d1-metrics-certs\") pod \"maas-api-e2e-derive-e557ff8f-d9567988c-pfzgn\" (UID: \"7bb564b0-14dd-41e4-9cb5-419abfd099d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e557ff8f-d9567988c-pfzgn" Sep 12 02:00:17.597224 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:17.597182 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mz2v\" (UniqueName: \"kubernetes.io/projected/7bb564b0-14dd-41e4-9cb5-419abfd099d1-kube-api-access-7mz2v\") pod \"maas-api-e2e-derive-e557ff8f-d9567988c-pfzgn\" (UID: \"7bb564b0-14dd-41e4-9cb5-419abfd099d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e557ff8f-d9567988c-pfzgn" Sep 12 02:00:17.598642 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:17.598610 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:17.697747 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:17.697708 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mz2v\" (UniqueName: \"kubernetes.io/projected/7bb564b0-14dd-41e4-9cb5-419abfd099d1-kube-api-access-7mz2v\") pod \"maas-api-e2e-derive-e557ff8f-d9567988c-pfzgn\" (UID: \"7bb564b0-14dd-41e4-9cb5-419abfd099d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e557ff8f-d9567988c-pfzgn" Sep 12 02:00:17.697936 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:17.697775 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bb564b0-14dd-41e4-9cb5-419abfd099d1-openshift-service-ca\") pod \"maas-api-e2e-derive-e557ff8f-d9567988c-pfzgn\" (UID: \"7bb564b0-14dd-41e4-9cb5-419abfd099d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e557ff8f-d9567988c-pfzgn" Sep 12 02:00:17.697936 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:17.697824 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bb564b0-14dd-41e4-9cb5-419abfd099d1-maas-api-tls\") pod \"maas-api-e2e-derive-e557ff8f-d9567988c-pfzgn\" (UID: \"7bb564b0-14dd-41e4-9cb5-419abfd099d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e557ff8f-d9567988c-pfzgn" Sep 12 02:00:17.697936 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:17.697869 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bb564b0-14dd-41e4-9cb5-419abfd099d1-metrics-certs\") pod \"maas-api-e2e-derive-e557ff8f-d9567988c-pfzgn\" (UID: \"7bb564b0-14dd-41e4-9cb5-419abfd099d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e557ff8f-d9567988c-pfzgn" Sep 12 02:00:17.698074 ip-10-0-133-79 kubenswrapper[2579]: E0912 02:00:17.697956 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-e557ff8f-serving-cert: secret "maas-api-e2e-derive-e557ff8f-serving-cert" not found Sep 12 02:00:17.698074 ip-10-0-133-79 kubenswrapper[2579]: E0912 02:00:17.698053 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bb564b0-14dd-41e4-9cb5-419abfd099d1-maas-api-tls podName:7bb564b0-14dd-41e4-9cb5-419abfd099d1 nodeName:}" failed. No retries permitted until 2026-09-12 02:00:18.198031815 +0000 UTC m=+920.990025587 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7bb564b0-14dd-41e4-9cb5-419abfd099d1-maas-api-tls") pod "maas-api-e2e-derive-e557ff8f-d9567988c-pfzgn" (UID: "7bb564b0-14dd-41e4-9cb5-419abfd099d1") : secret "maas-api-e2e-derive-e557ff8f-serving-cert" not found Sep 12 02:00:17.698647 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:17.698577 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bb564b0-14dd-41e4-9cb5-419abfd099d1-openshift-service-ca\") pod \"maas-api-e2e-derive-e557ff8f-d9567988c-pfzgn\" (UID: \"7bb564b0-14dd-41e4-9cb5-419abfd099d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e557ff8f-d9567988c-pfzgn" Sep 12 02:00:17.700412 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:17.700394 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bb564b0-14dd-41e4-9cb5-419abfd099d1-metrics-certs\") pod \"maas-api-e2e-derive-e557ff8f-d9567988c-pfzgn\" (UID: \"7bb564b0-14dd-41e4-9cb5-419abfd099d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e557ff8f-d9567988c-pfzgn" Sep 12 02:00:17.712531 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:17.712507 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mz2v\" (UniqueName: \"kubernetes.io/projected/7bb564b0-14dd-41e4-9cb5-419abfd099d1-kube-api-access-7mz2v\") pod \"maas-api-e2e-derive-e557ff8f-d9567988c-pfzgn\" (UID: \"7bb564b0-14dd-41e4-9cb5-419abfd099d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e557ff8f-d9567988c-pfzgn" Sep 12 02:00:17.963266 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:17.963218 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:18.052372 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:18.052334 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e557ff8f-7b54b979bb-7h85t"] Sep 12 02:00:18.052372 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:18.052364 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:18.056498 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:18.056474 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-e557ff8f-7b54b979bb-7h85t" Sep 12 02:00:18.058906 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:18.058886 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-e557ff8f\"" Sep 12 02:00:18.058999 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:18.058894 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-e557ff8f-dockercfg-7w6rq\"" Sep 12 02:00:18.066294 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:18.066268 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e557ff8f-7b54b979bb-7h85t"] Sep 12 02:00:18.132747 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:18.132718 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:18.203650 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:18.203614 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmtqh\" (UniqueName: \"kubernetes.io/projected/efadc538-bae2-4542-a190-02beaf890886-kube-api-access-zmtqh\") pod \"payload-pre-processing-e2e-derive-e557ff8f-7b54b979bb-7h85t\" (UID: \"efadc538-bae2-4542-a190-02beaf890886\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e557ff8f-7b54b979bb-7h85t" Sep 12 02:00:18.203822 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:18.203658 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efadc538-bae2-4542-a190-02beaf890886-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-e557ff8f-7b54b979bb-7h85t\" (UID: \"efadc538-bae2-4542-a190-02beaf890886\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e557ff8f-7b54b979bb-7h85t" Sep 12 02:00:18.203822 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:18.203707 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bb564b0-14dd-41e4-9cb5-419abfd099d1-maas-api-tls\") pod \"maas-api-e2e-derive-e557ff8f-d9567988c-pfzgn\" (UID: \"7bb564b0-14dd-41e4-9cb5-419abfd099d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e557ff8f-d9567988c-pfzgn" Sep 12 02:00:18.206275 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:18.206243 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bb564b0-14dd-41e4-9cb5-419abfd099d1-maas-api-tls\") pod \"maas-api-e2e-derive-e557ff8f-d9567988c-pfzgn\" (UID: \"7bb564b0-14dd-41e4-9cb5-419abfd099d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e557ff8f-d9567988c-pfzgn" Sep 12 02:00:18.304511 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:18.304391 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmtqh\" (UniqueName: \"kubernetes.io/projected/efadc538-bae2-4542-a190-02beaf890886-kube-api-access-zmtqh\") pod \"payload-pre-processing-e2e-derive-e557ff8f-7b54b979bb-7h85t\" (UID: \"efadc538-bae2-4542-a190-02beaf890886\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e557ff8f-7b54b979bb-7h85t" Sep 12 02:00:18.304511 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:18.304435 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efadc538-bae2-4542-a190-02beaf890886-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-e557ff8f-7b54b979bb-7h85t\" (UID: \"efadc538-bae2-4542-a190-02beaf890886\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e557ff8f-7b54b979bb-7h85t" Sep 12 02:00:18.305103 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:18.305084 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efadc538-bae2-4542-a190-02beaf890886-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-e557ff8f-7b54b979bb-7h85t\" (UID: \"efadc538-bae2-4542-a190-02beaf890886\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e557ff8f-7b54b979bb-7h85t" Sep 12 02:00:18.318181 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:18.318154 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmtqh\" (UniqueName: \"kubernetes.io/projected/efadc538-bae2-4542-a190-02beaf890886-kube-api-access-zmtqh\") pod \"payload-pre-processing-e2e-derive-e557ff8f-7b54b979bb-7h85t\" (UID: \"efadc538-bae2-4542-a190-02beaf890886\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e557ff8f-7b54b979bb-7h85t" Sep 12 02:00:18.368536 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:18.368501 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-e557ff8f-7b54b979bb-7h85t" Sep 12 02:00:18.380598 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:18.380577 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e557ff8f-d9567988c-pfzgn" Sep 12 02:00:18.643191 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:18.643141 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e557ff8f-7b54b979bb-7h85t"] Sep 12 02:00:18.645079 ip-10-0-133-79 kubenswrapper[2579]: W0912 02:00:18.645052 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefadc538_bae2_4542_a190_02beaf890886.slice/crio-e0aa44965540b3f781397ac71a373bbd8257d539490aee7a89739b9ae6845d21 WatchSource:0}: Error finding container e0aa44965540b3f781397ac71a373bbd8257d539490aee7a89739b9ae6845d21: Status 404 returned error can't find the container with id e0aa44965540b3f781397ac71a373bbd8257d539490aee7a89739b9ae6845d21 Sep 12 02:00:18.663386 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:18.663361 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e557ff8f-d9567988c-pfzgn"] Sep 12 02:00:18.665325 ip-10-0-133-79 kubenswrapper[2579]: W0912 02:00:18.665279 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bb564b0_14dd_41e4_9cb5_419abfd099d1.slice/crio-37723fd693c644087683374503ce8077298992b3f521725a4d585e58c2551adf WatchSource:0}: Error finding container 37723fd693c644087683374503ce8077298992b3f521725a4d585e58c2551adf: Status 404 returned error can't find the container with id 37723fd693c644087683374503ce8077298992b3f521725a4d585e58c2551adf Sep 12 02:00:19.321538 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:19.321417 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e557ff8f-d9567988c-pfzgn" event={"ID":"7bb564b0-14dd-41e4-9cb5-419abfd099d1","Type":"ContainerStarted","Data":"51bd6e4304feb1e3003e9ccc3d43afc7209fead7622a3d16bf30ea8511603e1b"} Sep 12 02:00:19.321538 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:19.321514 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e557ff8f-d9567988c-pfzgn" event={"ID":"7bb564b0-14dd-41e4-9cb5-419abfd099d1","Type":"ContainerStarted","Data":"37723fd693c644087683374503ce8077298992b3f521725a4d585e58c2551adf"} Sep 12 02:00:19.321783 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:19.321622 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e557ff8f-d9567988c-pfzgn" Sep 12 02:00:19.323053 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:19.323031 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-e557ff8f-7b54b979bb-7h85t" event={"ID":"efadc538-bae2-4542-a190-02beaf890886","Type":"ContainerStarted","Data":"a16ba448b65fabcf05eca954bfeae9a4c182cafdae869db08b592efa6455205a"} Sep 12 02:00:19.323165 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:19.323067 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-e557ff8f-7b54b979bb-7h85t" event={"ID":"efadc538-bae2-4542-a190-02beaf890886","Type":"ContainerStarted","Data":"e0aa44965540b3f781397ac71a373bbd8257d539490aee7a89739b9ae6845d21"} Sep 12 02:00:19.323165 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:19.323163 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-e557ff8f-7b54b979bb-7h85t" Sep 12 02:00:19.339877 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:19.339822 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e557ff8f-d9567988c-pfzgn" podStartSLOduration=1.978436944 podStartE2EDuration="2.339805921s" totalImagesPullingTime="361.368977ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:00:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:00:19.338135906 +0000 UTC m=+922.130129701" watchObservedRunningTime="2026-09-12 02:00:19.339805921 +0000 UTC m=+922.131799733" Sep 12 02:00:19.354604 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:19.354551 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-e557ff8f-7b54b979bb-7h85t" podStartSLOduration=1.354531582 podStartE2EDuration="1.354531582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:00:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:00:19.352930097 +0000 UTC m=+922.144923911" watchObservedRunningTime="2026-09-12 02:00:19.354531582 +0000 UTC m=+922.146525378" Sep 12 02:00:25.335407 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:25.335378 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e557ff8f-d9567988c-pfzgn" Sep 12 02:00:30.332624 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:30.332593 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-e557ff8f-7b54b979bb-7h85t" Sep 12 02:00:34.234132 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:34.234097 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:36.926165 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:36.926131 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:00:39.224174 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:39.224127 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:40.594251 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:40.594215 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:41.253486 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:41.253437 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:41.584613 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:41.584522 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:41.684773 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:41.684737 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:41.960359 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:41.960322 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:42.616609 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:42.616575 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:43.357245 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:43.357206 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:44.066142 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:44.066114 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:44.791323 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:44.791276 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:45.558098 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:45.558063 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:46.368729 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:46.368687 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:47.037166 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:47.037127 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:47.741809 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:47.741771 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:48.223433 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:48.223399 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-3fc028/test-model-3fc028-kserve-556bc88545-wt94k"] Sep 12 02:00:48.227515 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:48.227496 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3fc028/test-model-3fc028-kserve-556bc88545-wt94k" Sep 12 02:00:48.230810 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:48.230792 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-3fc028\"/\"default-dockercfg-76ql5\"" Sep 12 02:00:48.231097 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:48.230792 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-3fc028\"/\"test-model-3fc028-kserve-self-signed-certs\"" Sep 12 02:00:48.231281 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:48.230822 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-3fc028\"/\"kube-root-ca.crt\"" Sep 12 02:00:48.231800 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:48.230842 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-3fc028\"/\"openshift-service-ca.crt\"" Sep 12 02:00:48.234761 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:48.234738 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-3fc028/test-model-3fc028-kserve-556bc88545-wt94k"] Sep 12 02:00:48.252267 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:48.252238 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:48.303642 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:48.303611 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/127d7708-4a3f-4ad1-b90f-bf552a9a0afb-tls-certs\") pod \"test-model-3fc028-kserve-556bc88545-wt94k\" (UID: \"127d7708-4a3f-4ad1-b90f-bf552a9a0afb\") " pod="ai-tenant-e2e-inf-a-3fc028/test-model-3fc028-kserve-556bc88545-wt94k" Sep 12 02:00:48.303840 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:48.303655 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psmzs\" (UniqueName: \"kubernetes.io/projected/127d7708-4a3f-4ad1-b90f-bf552a9a0afb-kube-api-access-psmzs\") pod \"test-model-3fc028-kserve-556bc88545-wt94k\" (UID: \"127d7708-4a3f-4ad1-b90f-bf552a9a0afb\") " pod="ai-tenant-e2e-inf-a-3fc028/test-model-3fc028-kserve-556bc88545-wt94k" Sep 12 02:00:48.303840 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:48.303707 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/127d7708-4a3f-4ad1-b90f-bf552a9a0afb-dshm\") pod \"test-model-3fc028-kserve-556bc88545-wt94k\" (UID: \"127d7708-4a3f-4ad1-b90f-bf552a9a0afb\") " pod="ai-tenant-e2e-inf-a-3fc028/test-model-3fc028-kserve-556bc88545-wt94k" Sep 12 02:00:48.303840 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:48.303745 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/127d7708-4a3f-4ad1-b90f-bf552a9a0afb-model-cache\") pod \"test-model-3fc028-kserve-556bc88545-wt94k\" (UID: \"127d7708-4a3f-4ad1-b90f-bf552a9a0afb\") " pod="ai-tenant-e2e-inf-a-3fc028/test-model-3fc028-kserve-556bc88545-wt94k" Sep 12 02:00:48.303840 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:48.303815 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/127d7708-4a3f-4ad1-b90f-bf552a9a0afb-home\") pod \"test-model-3fc028-kserve-556bc88545-wt94k\" (UID: \"127d7708-4a3f-4ad1-b90f-bf552a9a0afb\") " pod="ai-tenant-e2e-inf-a-3fc028/test-model-3fc028-kserve-556bc88545-wt94k" Sep 12 02:00:48.304024 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:48.303865 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/127d7708-4a3f-4ad1-b90f-bf552a9a0afb-tmp-dir\") pod \"test-model-3fc028-kserve-556bc88545-wt94k\" (UID: \"127d7708-4a3f-4ad1-b90f-bf552a9a0afb\") " pod="ai-tenant-e2e-inf-a-3fc028/test-model-3fc028-kserve-556bc88545-wt94k" Sep 12 02:00:48.404473 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:48.404407 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psmzs\" (UniqueName: \"kubernetes.io/projected/127d7708-4a3f-4ad1-b90f-bf552a9a0afb-kube-api-access-psmzs\") pod \"test-model-3fc028-kserve-556bc88545-wt94k\" (UID: \"127d7708-4a3f-4ad1-b90f-bf552a9a0afb\") " pod="ai-tenant-e2e-inf-a-3fc028/test-model-3fc028-kserve-556bc88545-wt94k" Sep 12 02:00:48.404651 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:48.404492 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/127d7708-4a3f-4ad1-b90f-bf552a9a0afb-dshm\") pod \"test-model-3fc028-kserve-556bc88545-wt94k\" (UID: \"127d7708-4a3f-4ad1-b90f-bf552a9a0afb\") " pod="ai-tenant-e2e-inf-a-3fc028/test-model-3fc028-kserve-556bc88545-wt94k" Sep 12 02:00:48.404651 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:48.404533 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/127d7708-4a3f-4ad1-b90f-bf552a9a0afb-model-cache\") pod \"test-model-3fc028-kserve-556bc88545-wt94k\" (UID: \"127d7708-4a3f-4ad1-b90f-bf552a9a0afb\") " pod="ai-tenant-e2e-inf-a-3fc028/test-model-3fc028-kserve-556bc88545-wt94k" Sep 12 02:00:48.404651 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:48.404562 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/127d7708-4a3f-4ad1-b90f-bf552a9a0afb-home\") pod \"test-model-3fc028-kserve-556bc88545-wt94k\" (UID: \"127d7708-4a3f-4ad1-b90f-bf552a9a0afb\") " pod="ai-tenant-e2e-inf-a-3fc028/test-model-3fc028-kserve-556bc88545-wt94k" Sep 12 02:00:48.404651 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:48.404593 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/127d7708-4a3f-4ad1-b90f-bf552a9a0afb-tmp-dir\") pod \"test-model-3fc028-kserve-556bc88545-wt94k\" (UID: \"127d7708-4a3f-4ad1-b90f-bf552a9a0afb\") " pod="ai-tenant-e2e-inf-a-3fc028/test-model-3fc028-kserve-556bc88545-wt94k" Sep 12 02:00:48.404853 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:48.404673 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/127d7708-4a3f-4ad1-b90f-bf552a9a0afb-tls-certs\") pod \"test-model-3fc028-kserve-556bc88545-wt94k\" (UID: \"127d7708-4a3f-4ad1-b90f-bf552a9a0afb\") " pod="ai-tenant-e2e-inf-a-3fc028/test-model-3fc028-kserve-556bc88545-wt94k" Sep 12 02:00:48.404961 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:48.404942 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/127d7708-4a3f-4ad1-b90f-bf552a9a0afb-home\") pod \"test-model-3fc028-kserve-556bc88545-wt94k\" (UID: \"127d7708-4a3f-4ad1-b90f-bf552a9a0afb\") " pod="ai-tenant-e2e-inf-a-3fc028/test-model-3fc028-kserve-556bc88545-wt94k" Sep 12 02:00:48.405034 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:48.404991 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/127d7708-4a3f-4ad1-b90f-bf552a9a0afb-model-cache\") pod \"test-model-3fc028-kserve-556bc88545-wt94k\" (UID: \"127d7708-4a3f-4ad1-b90f-bf552a9a0afb\") " pod="ai-tenant-e2e-inf-a-3fc028/test-model-3fc028-kserve-556bc88545-wt94k" Sep 12 02:00:48.405034 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:48.405020 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/127d7708-4a3f-4ad1-b90f-bf552a9a0afb-tmp-dir\") pod \"test-model-3fc028-kserve-556bc88545-wt94k\" (UID: \"127d7708-4a3f-4ad1-b90f-bf552a9a0afb\") " pod="ai-tenant-e2e-inf-a-3fc028/test-model-3fc028-kserve-556bc88545-wt94k" Sep 12 02:00:48.407003 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:48.406980 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/127d7708-4a3f-4ad1-b90f-bf552a9a0afb-dshm\") pod \"test-model-3fc028-kserve-556bc88545-wt94k\" (UID: \"127d7708-4a3f-4ad1-b90f-bf552a9a0afb\") " pod="ai-tenant-e2e-inf-a-3fc028/test-model-3fc028-kserve-556bc88545-wt94k" Sep 12 02:00:48.407268 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:48.407248 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/127d7708-4a3f-4ad1-b90f-bf552a9a0afb-tls-certs\") pod \"test-model-3fc028-kserve-556bc88545-wt94k\" (UID: \"127d7708-4a3f-4ad1-b90f-bf552a9a0afb\") " pod="ai-tenant-e2e-inf-a-3fc028/test-model-3fc028-kserve-556bc88545-wt94k" Sep 12 02:00:48.412584 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:48.412563 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psmzs\" (UniqueName: \"kubernetes.io/projected/127d7708-4a3f-4ad1-b90f-bf552a9a0afb-kube-api-access-psmzs\") pod \"test-model-3fc028-kserve-556bc88545-wt94k\" (UID: \"127d7708-4a3f-4ad1-b90f-bf552a9a0afb\") " pod="ai-tenant-e2e-inf-a-3fc028/test-model-3fc028-kserve-556bc88545-wt94k" Sep 12 02:00:48.539490 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:48.539371 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3fc028/test-model-3fc028-kserve-556bc88545-wt94k" Sep 12 02:00:48.600663 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:48.600609 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:48.813130 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:48.813108 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-3fc028/test-model-3fc028-kserve-556bc88545-wt94k"] Sep 12 02:00:48.815749 ip-10-0-133-79 kubenswrapper[2579]: W0912 02:00:48.815722 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod127d7708_4a3f_4ad1_b90f_bf552a9a0afb.slice/crio-6452dddad8f6af369afc21692d58a089b161ef64bd89c36575bd9c1f85eab411 WatchSource:0}: Error finding container 6452dddad8f6af369afc21692d58a089b161ef64bd89c36575bd9c1f85eab411: Status 404 returned error can't find the container with id 6452dddad8f6af369afc21692d58a089b161ef64bd89c36575bd9c1f85eab411 Sep 12 02:00:49.124223 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:49.124177 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:49.489267 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:49.489181 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3fc028/test-model-3fc028-kserve-556bc88545-wt94k" event={"ID":"127d7708-4a3f-4ad1-b90f-bf552a9a0afb","Type":"ContainerStarted","Data":"5dd7ace081fbd5524a8e92f7c6a3d73d19621d48a703dd4f1ca67450efbb899f"} Sep 12 02:00:49.489267 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:49.489225 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3fc028/test-model-3fc028-kserve-556bc88545-wt94k" event={"ID":"127d7708-4a3f-4ad1-b90f-bf552a9a0afb","Type":"ContainerStarted","Data":"6452dddad8f6af369afc21692d58a089b161ef64bd89c36575bd9c1f85eab411"} Sep 12 02:00:49.505678 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:49.505626 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-3fc028/test-model-3fc028-kserve-556bc88545-wt94k" podStartSLOduration=1.505607307 podStartE2EDuration="1.505607307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:00:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:00:49.503680172 +0000 UTC m=+952.295673967" watchObservedRunningTime="2026-09-12 02:00:49.505607307 +0000 UTC m=+952.297601102" Sep 12 02:00:49.823825 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:49.823746 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:50.539144 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:50.539102 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:54.138014 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:54.137968 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:54.662475 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:54.662425 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e557ff8f-d9567988c-pfzgn"] Sep 12 02:00:54.662756 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:54.662726 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e557ff8f-d9567988c-pfzgn" podUID="7bb564b0-14dd-41e4-9cb5-419abfd099d1" containerName="maas-api" containerID="cri-o://51bd6e4304feb1e3003e9ccc3d43afc7209fead7622a3d16bf30ea8511603e1b" gracePeriod=30 Sep 12 02:00:54.692154 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:54.692119 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:54.737098 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:54.737055 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e557ff8f-7b54b979bb-7h85t"] Sep 12 02:00:54.737357 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:54.737330 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-e557ff8f-7b54b979bb-7h85t" podUID="efadc538-bae2-4542-a190-02beaf890886" containerName="payload-pre-processing" containerID="cri-o://a16ba448b65fabcf05eca954bfeae9a4c182cafdae869db08b592efa6455205a" gracePeriod=30 Sep 12 02:00:54.793349 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:54.793305 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:54.881847 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:54.881805 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:54.963999 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:54.963963 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:55.028160 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:55.028118 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e557ff8f-d9567988c-pfzgn" Sep 12 02:00:55.031390 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:55.031362 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-e557ff8f-7b54b979bb-7h85t" Sep 12 02:00:55.039799 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:55.039763 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:55.173368 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:55.173327 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mz2v\" (UniqueName: \"kubernetes.io/projected/7bb564b0-14dd-41e4-9cb5-419abfd099d1-kube-api-access-7mz2v\") pod \"7bb564b0-14dd-41e4-9cb5-419abfd099d1\" (UID: \"7bb564b0-14dd-41e4-9cb5-419abfd099d1\") " Sep 12 02:00:55.173368 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:55.173370 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bb564b0-14dd-41e4-9cb5-419abfd099d1-openshift-service-ca\") pod \"7bb564b0-14dd-41e4-9cb5-419abfd099d1\" (UID: \"7bb564b0-14dd-41e4-9cb5-419abfd099d1\") " Sep 12 02:00:55.173946 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:55.173388 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bb564b0-14dd-41e4-9cb5-419abfd099d1-maas-api-tls\") pod \"7bb564b0-14dd-41e4-9cb5-419abfd099d1\" (UID: \"7bb564b0-14dd-41e4-9cb5-419abfd099d1\") " Sep 12 02:00:55.173946 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:55.173413 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bb564b0-14dd-41e4-9cb5-419abfd099d1-metrics-certs\") pod \"7bb564b0-14dd-41e4-9cb5-419abfd099d1\" (UID: \"7bb564b0-14dd-41e4-9cb5-419abfd099d1\") " Sep 12 02:00:55.173946 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:55.173597 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmtqh\" (UniqueName: \"kubernetes.io/projected/efadc538-bae2-4542-a190-02beaf890886-kube-api-access-zmtqh\") pod \"efadc538-bae2-4542-a190-02beaf890886\" (UID: \"efadc538-bae2-4542-a190-02beaf890886\") " Sep 12 02:00:55.173946 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:55.173658 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efadc538-bae2-4542-a190-02beaf890886-plugins-config-volume\") pod \"efadc538-bae2-4542-a190-02beaf890886\" (UID: \"efadc538-bae2-4542-a190-02beaf890886\") " Sep 12 02:00:55.173946 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:55.173861 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bb564b0-14dd-41e4-9cb5-419abfd099d1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7bb564b0-14dd-41e4-9cb5-419abfd099d1" (UID: "7bb564b0-14dd-41e4-9cb5-419abfd099d1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 12 02:00:55.174386 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:55.174093 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efadc538-bae2-4542-a190-02beaf890886-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "efadc538-bae2-4542-a190-02beaf890886" (UID: "efadc538-bae2-4542-a190-02beaf890886"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 12 02:00:55.174386 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:55.174161 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bb564b0-14dd-41e4-9cb5-419abfd099d1-openshift-service-ca\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:00:55.175805 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:55.175780 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bb564b0-14dd-41e4-9cb5-419abfd099d1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7bb564b0-14dd-41e4-9cb5-419abfd099d1" (UID: "7bb564b0-14dd-41e4-9cb5-419abfd099d1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 12 02:00:55.175921 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:55.175822 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bb564b0-14dd-41e4-9cb5-419abfd099d1-kube-api-access-7mz2v" (OuterVolumeSpecName: "kube-api-access-7mz2v") pod "7bb564b0-14dd-41e4-9cb5-419abfd099d1" (UID: "7bb564b0-14dd-41e4-9cb5-419abfd099d1"). InnerVolumeSpecName "kube-api-access-7mz2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 12 02:00:55.176274 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:55.176253 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bb564b0-14dd-41e4-9cb5-419abfd099d1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7bb564b0-14dd-41e4-9cb5-419abfd099d1" (UID: "7bb564b0-14dd-41e4-9cb5-419abfd099d1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 12 02:00:55.176371 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:55.176351 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efadc538-bae2-4542-a190-02beaf890886-kube-api-access-zmtqh" (OuterVolumeSpecName: "kube-api-access-zmtqh") pod "efadc538-bae2-4542-a190-02beaf890886" (UID: "efadc538-bae2-4542-a190-02beaf890886"). InnerVolumeSpecName "kube-api-access-zmtqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 12 02:00:55.275813 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:55.275729 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmtqh\" (UniqueName: \"kubernetes.io/projected/efadc538-bae2-4542-a190-02beaf890886-kube-api-access-zmtqh\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:00:55.275813 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:55.275758 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efadc538-bae2-4542-a190-02beaf890886-plugins-config-volume\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:00:55.275813 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:55.275768 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mz2v\" (UniqueName: \"kubernetes.io/projected/7bb564b0-14dd-41e4-9cb5-419abfd099d1-kube-api-access-7mz2v\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:00:55.275813 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:55.275780 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bb564b0-14dd-41e4-9cb5-419abfd099d1-maas-api-tls\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:00:55.275813 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:55.275789 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bb564b0-14dd-41e4-9cb5-419abfd099d1-metrics-certs\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:00:55.374754 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:55.374714 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:55.523033 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:55.522995 2579 generic.go:356] "Generic (PLEG): container finished" podID="7bb564b0-14dd-41e4-9cb5-419abfd099d1" containerID="51bd6e4304feb1e3003e9ccc3d43afc7209fead7622a3d16bf30ea8511603e1b" exitCode=0 Sep 12 02:00:55.523216 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:55.523044 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e557ff8f-d9567988c-pfzgn" Sep 12 02:00:55.523216 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:55.523085 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e557ff8f-d9567988c-pfzgn" event={"ID":"7bb564b0-14dd-41e4-9cb5-419abfd099d1","Type":"ContainerDied","Data":"51bd6e4304feb1e3003e9ccc3d43afc7209fead7622a3d16bf30ea8511603e1b"} Sep 12 02:00:55.523216 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:55.523134 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e557ff8f-d9567988c-pfzgn" event={"ID":"7bb564b0-14dd-41e4-9cb5-419abfd099d1","Type":"ContainerDied","Data":"37723fd693c644087683374503ce8077298992b3f521725a4d585e58c2551adf"} Sep 12 02:00:55.523216 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:55.523164 2579 scope.go:117] "RemoveContainer" containerID="51bd6e4304feb1e3003e9ccc3d43afc7209fead7622a3d16bf30ea8511603e1b" Sep 12 02:00:55.524658 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:55.524637 2579 generic.go:356] "Generic (PLEG): container finished" podID="efadc538-bae2-4542-a190-02beaf890886" containerID="a16ba448b65fabcf05eca954bfeae9a4c182cafdae869db08b592efa6455205a" exitCode=0 Sep 12 02:00:55.524925 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:55.524674 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-e557ff8f-7b54b979bb-7h85t" event={"ID":"efadc538-bae2-4542-a190-02beaf890886","Type":"ContainerDied","Data":"a16ba448b65fabcf05eca954bfeae9a4c182cafdae869db08b592efa6455205a"} Sep 12 02:00:55.524925 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:55.524685 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-e557ff8f-7b54b979bb-7h85t" Sep 12 02:00:55.524925 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:55.524710 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-e557ff8f-7b54b979bb-7h85t" event={"ID":"efadc538-bae2-4542-a190-02beaf890886","Type":"ContainerDied","Data":"e0aa44965540b3f781397ac71a373bbd8257d539490aee7a89739b9ae6845d21"} Sep 12 02:00:55.533470 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:55.533435 2579 scope.go:117] "RemoveContainer" containerID="51bd6e4304feb1e3003e9ccc3d43afc7209fead7622a3d16bf30ea8511603e1b" Sep 12 02:00:55.533794 ip-10-0-133-79 kubenswrapper[2579]: E0912 02:00:55.533773 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51bd6e4304feb1e3003e9ccc3d43afc7209fead7622a3d16bf30ea8511603e1b\": container with ID starting with 51bd6e4304feb1e3003e9ccc3d43afc7209fead7622a3d16bf30ea8511603e1b not found: ID does not exist" containerID="51bd6e4304feb1e3003e9ccc3d43afc7209fead7622a3d16bf30ea8511603e1b" Sep 12 02:00:55.533903 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:55.533802 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51bd6e4304feb1e3003e9ccc3d43afc7209fead7622a3d16bf30ea8511603e1b"} err="failed to get container status \"51bd6e4304feb1e3003e9ccc3d43afc7209fead7622a3d16bf30ea8511603e1b\": rpc error: code = NotFound desc = could not find container \"51bd6e4304feb1e3003e9ccc3d43afc7209fead7622a3d16bf30ea8511603e1b\": container with ID starting with 51bd6e4304feb1e3003e9ccc3d43afc7209fead7622a3d16bf30ea8511603e1b not found: ID does not exist" Sep 12 02:00:55.533903 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:55.533821 2579 scope.go:117] "RemoveContainer" containerID="a16ba448b65fabcf05eca954bfeae9a4c182cafdae869db08b592efa6455205a" Sep 12 02:00:55.542685 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:55.542668 2579 scope.go:117] "RemoveContainer" containerID="a16ba448b65fabcf05eca954bfeae9a4c182cafdae869db08b592efa6455205a" Sep 12 02:00:55.542994 ip-10-0-133-79 kubenswrapper[2579]: E0912 02:00:55.542945 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a16ba448b65fabcf05eca954bfeae9a4c182cafdae869db08b592efa6455205a\": container with ID starting with a16ba448b65fabcf05eca954bfeae9a4c182cafdae869db08b592efa6455205a not found: ID does not exist" containerID="a16ba448b65fabcf05eca954bfeae9a4c182cafdae869db08b592efa6455205a" Sep 12 02:00:55.542994 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:55.542973 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a16ba448b65fabcf05eca954bfeae9a4c182cafdae869db08b592efa6455205a"} err="failed to get container status \"a16ba448b65fabcf05eca954bfeae9a4c182cafdae869db08b592efa6455205a\": rpc error: code = NotFound desc = could not find container \"a16ba448b65fabcf05eca954bfeae9a4c182cafdae869db08b592efa6455205a\": container with ID starting with a16ba448b65fabcf05eca954bfeae9a4c182cafdae869db08b592efa6455205a not found: ID does not exist" Sep 12 02:00:55.548675 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:55.548644 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e557ff8f-7b54b979bb-7h85t"] Sep 12 02:00:55.552764 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:55.552731 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e557ff8f-7b54b979bb-7h85t"] Sep 12 02:00:55.569885 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:55.569852 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e557ff8f-d9567988c-pfzgn"] Sep 12 02:00:55.573789 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:55.573765 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e557ff8f-d9567988c-pfzgn"] Sep 12 02:00:55.727148 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:55.727115 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bb564b0-14dd-41e4-9cb5-419abfd099d1" path="/var/lib/kubelet/pods/7bb564b0-14dd-41e4-9cb5-419abfd099d1/volumes" Sep 12 02:00:55.727610 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:55.727590 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efadc538-bae2-4542-a190-02beaf890886" path="/var/lib/kubelet/pods/efadc538-bae2-4542-a190-02beaf890886/volumes" Sep 12 02:00:55.735625 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:55.735600 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:55.829526 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:55.829425 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:55.928853 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:55.928822 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:56.132297 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:56.132261 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:56.700668 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:56.700631 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-9ef13e-7f6b799448-rmszj"] Sep 12 02:00:56.701079 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:56.700862 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9ef13e-7f6b799448-rmszj" podUID="ca57934c-35fb-4cac-8c96-e3861edbb214" containerName="maas-api" containerID="cri-o://6c15699cc377180d2a6cccae0cac8152d2086b39b4878db3f8351ceb39bfa926" gracePeriod=30 Sep 12 02:00:56.741813 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:56.741755 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:56.842682 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:56.842648 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:56.943380 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:56.943335 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:56.970263 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:56.970238 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9ef13e-7f6b799448-rmszj" Sep 12 02:00:57.044295 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:57.044264 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:57.096424 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:57.096389 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca57934c-35fb-4cac-8c96-e3861edbb214-openshift-service-ca\") pod \"ca57934c-35fb-4cac-8c96-e3861edbb214\" (UID: \"ca57934c-35fb-4cac-8c96-e3861edbb214\") " Sep 12 02:00:57.096641 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:57.096547 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca57934c-35fb-4cac-8c96-e3861edbb214-maas-api-tls\") pod \"ca57934c-35fb-4cac-8c96-e3861edbb214\" (UID: \"ca57934c-35fb-4cac-8c96-e3861edbb214\") " Sep 12 02:00:57.096641 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:57.096580 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7rvmv\" (UniqueName: \"kubernetes.io/projected/ca57934c-35fb-4cac-8c96-e3861edbb214-kube-api-access-7rvmv\") pod \"ca57934c-35fb-4cac-8c96-e3861edbb214\" (UID: \"ca57934c-35fb-4cac-8c96-e3861edbb214\") " Sep 12 02:00:57.096764 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:57.096729 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca57934c-35fb-4cac-8c96-e3861edbb214-metrics-certs\") pod \"ca57934c-35fb-4cac-8c96-e3861edbb214\" (UID: \"ca57934c-35fb-4cac-8c96-e3861edbb214\") " Sep 12 02:00:57.096764 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:57.096754 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca57934c-35fb-4cac-8c96-e3861edbb214-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ca57934c-35fb-4cac-8c96-e3861edbb214" (UID: "ca57934c-35fb-4cac-8c96-e3861edbb214"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 12 02:00:57.097012 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:57.096995 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca57934c-35fb-4cac-8c96-e3861edbb214-openshift-service-ca\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:00:57.098839 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:57.098810 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca57934c-35fb-4cac-8c96-e3861edbb214-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ca57934c-35fb-4cac-8c96-e3861edbb214" (UID: "ca57934c-35fb-4cac-8c96-e3861edbb214"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 12 02:00:57.098951 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:57.098861 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca57934c-35fb-4cac-8c96-e3861edbb214-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ca57934c-35fb-4cac-8c96-e3861edbb214" (UID: "ca57934c-35fb-4cac-8c96-e3861edbb214"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 12 02:00:57.098951 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:57.098924 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca57934c-35fb-4cac-8c96-e3861edbb214-kube-api-access-7rvmv" (OuterVolumeSpecName: "kube-api-access-7rvmv") pod "ca57934c-35fb-4cac-8c96-e3861edbb214" (UID: "ca57934c-35fb-4cac-8c96-e3861edbb214"). InnerVolumeSpecName "kube-api-access-7rvmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 12 02:00:57.132671 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:57.132630 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:57.198224 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:57.198190 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca57934c-35fb-4cac-8c96-e3861edbb214-metrics-certs\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:00:57.198224 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:57.198219 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca57934c-35fb-4cac-8c96-e3861edbb214-maas-api-tls\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:00:57.198224 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:57.198227 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rvmv\" (UniqueName: \"kubernetes.io/projected/ca57934c-35fb-4cac-8c96-e3861edbb214-kube-api-access-7rvmv\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:00:57.267634 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:57.267524 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:57.358788 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:57.358751 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:57.506517 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:57.506476 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:57.538753 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:57.538671 2579 generic.go:356] "Generic (PLEG): container finished" podID="ca57934c-35fb-4cac-8c96-e3861edbb214" containerID="6c15699cc377180d2a6cccae0cac8152d2086b39b4878db3f8351ceb39bfa926" exitCode=0 Sep 12 02:00:57.538753 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:57.538717 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9ef13e-7f6b799448-rmszj" Sep 12 02:00:57.538937 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:57.538785 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9ef13e-7f6b799448-rmszj" event={"ID":"ca57934c-35fb-4cac-8c96-e3861edbb214","Type":"ContainerDied","Data":"6c15699cc377180d2a6cccae0cac8152d2086b39b4878db3f8351ceb39bfa926"} Sep 12 02:00:57.538937 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:57.538837 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9ef13e-7f6b799448-rmszj" event={"ID":"ca57934c-35fb-4cac-8c96-e3861edbb214","Type":"ContainerDied","Data":"0df5268f0dbf1f024625896e8baddf62d2de755912d857aa03948357a126978c"} Sep 12 02:00:57.538937 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:57.538867 2579 scope.go:117] "RemoveContainer" containerID="6c15699cc377180d2a6cccae0cac8152d2086b39b4878db3f8351ceb39bfa926" Sep 12 02:00:57.548957 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:57.548937 2579 scope.go:117] "RemoveContainer" containerID="6c15699cc377180d2a6cccae0cac8152d2086b39b4878db3f8351ceb39bfa926" Sep 12 02:00:57.549238 ip-10-0-133-79 kubenswrapper[2579]: E0912 02:00:57.549218 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c15699cc377180d2a6cccae0cac8152d2086b39b4878db3f8351ceb39bfa926\": container with ID starting with 6c15699cc377180d2a6cccae0cac8152d2086b39b4878db3f8351ceb39bfa926 not found: ID does not exist" containerID="6c15699cc377180d2a6cccae0cac8152d2086b39b4878db3f8351ceb39bfa926" Sep 12 02:00:57.549290 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:57.549247 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c15699cc377180d2a6cccae0cac8152d2086b39b4878db3f8351ceb39bfa926"} err="failed to get container status \"6c15699cc377180d2a6cccae0cac8152d2086b39b4878db3f8351ceb39bfa926\": rpc error: code = NotFound desc = could not find container \"6c15699cc377180d2a6cccae0cac8152d2086b39b4878db3f8351ceb39bfa926\": container with ID starting with 6c15699cc377180d2a6cccae0cac8152d2086b39b4878db3f8351ceb39bfa926 not found: ID does not exist" Sep 12 02:00:57.560662 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:57.560633 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-9ef13e-7f6b799448-rmszj"] Sep 12 02:00:57.566938 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:57.566909 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-9ef13e-7f6b799448-rmszj"] Sep 12 02:00:57.726823 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:57.726790 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca57934c-35fb-4cac-8c96-e3861edbb214" path="/var/lib/kubelet/pods/ca57934c-35fb-4cac-8c96-e3861edbb214/volumes" Sep 12 02:00:58.089172 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:58.089135 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:58.540036 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:58.540002 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-3fc028/test-model-3fc028-kserve-556bc88545-wt94k" Sep 12 02:00:58.540260 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:58.540049 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-3fc028/test-model-3fc028-kserve-556bc88545-wt94k" Sep 12 02:00:58.553113 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:58.553090 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-3fc028/test-model-3fc028-kserve-556bc88545-wt94k" Sep 12 02:00:58.860655 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:58.860564 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:59.561272 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:59.561242 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-3fc028/test-model-3fc028-kserve-556bc88545-wt94k" Sep 12 02:00:59.631641 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:59.631594 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:00:59.931838 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:00:59.931798 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:01:00.336372 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:00.336278 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:01:01.096505 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:01.096441 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:01:02.581092 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:02.581063 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:01:03.089353 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:03.089322 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:01:03.148034 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:03.148000 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:01:03.274043 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:03.273937 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:01:03.350799 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:03.350701 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:01:03.432440 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:03.432396 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:01:03.537253 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:03.537213 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:01:03.991765 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:03.991725 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:01:08.228342 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:08.228307 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:01:27.263143 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:27.263052 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:01:29.985179 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:29.985144 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:01:32.732041 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:32.732004 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:01:35.550046 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:35.549952 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:01:36.063177 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:36.063144 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:01:36.191960 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:36.191929 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:01:36.290132 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:36.290093 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:01:39.467608 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:39.467577 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:01:39.731785 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:39.731698 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:01:40.200069 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:40.200036 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:01:40.540525 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:40.540426 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:01:40.825400 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:40.825306 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:01:43.075899 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:43.075860 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:01:43.745253 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:43.745219 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:01:43.846217 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:43.846185 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:01:43.946434 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:43.946402 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:01:44.575787 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:44.575746 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:01:45.448954 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:45.448916 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:01:46.268542 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:46.268513 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:01:46.928179 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:46.928134 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b838b7-8588bdf7cd-xmndp"] Sep 12 02:01:46.928911 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:46.928882 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bb564b0-14dd-41e4-9cb5-419abfd099d1" containerName="maas-api" Sep 12 02:01:46.928911 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:46.928913 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bb564b0-14dd-41e4-9cb5-419abfd099d1" containerName="maas-api" Sep 12 02:01:46.929110 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:46.928952 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca57934c-35fb-4cac-8c96-e3861edbb214" containerName="maas-api" Sep 12 02:01:46.929110 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:46.928961 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca57934c-35fb-4cac-8c96-e3861edbb214" containerName="maas-api" Sep 12 02:01:46.929110 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:46.928971 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efadc538-bae2-4542-a190-02beaf890886" containerName="payload-pre-processing" Sep 12 02:01:46.929110 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:46.928980 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="efadc538-bae2-4542-a190-02beaf890886" containerName="payload-pre-processing" Sep 12 02:01:46.929110 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:46.929086 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca57934c-35fb-4cac-8c96-e3861edbb214" containerName="maas-api" Sep 12 02:01:46.929110 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:46.929105 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bb564b0-14dd-41e4-9cb5-419abfd099d1" containerName="maas-api" Sep 12 02:01:46.929339 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:46.929119 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="efadc538-bae2-4542-a190-02beaf890886" containerName="payload-pre-processing" Sep 12 02:01:46.932844 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:46.932825 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b838b7-8588bdf7cd-xmndp" Sep 12 02:01:46.935282 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:46.935257 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-b838b7-serving-cert\"" Sep 12 02:01:46.941794 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:46.941767 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b838b7-8588bdf7cd-xmndp"] Sep 12 02:01:46.962713 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:46.962666 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:01:46.997073 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:46.997034 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5971011-f9d7-428a-951c-7e1f8484578a-openshift-service-ca\") pod \"maas-api-e2e-explicit-b838b7-8588bdf7cd-xmndp\" (UID: \"d5971011-f9d7-428a-951c-7e1f8484578a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b838b7-8588bdf7cd-xmndp" Sep 12 02:01:46.997073 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:46.997072 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5971011-f9d7-428a-951c-7e1f8484578a-metrics-certs\") pod \"maas-api-e2e-explicit-b838b7-8588bdf7cd-xmndp\" (UID: \"d5971011-f9d7-428a-951c-7e1f8484578a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b838b7-8588bdf7cd-xmndp" Sep 12 02:01:46.997320 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:46.997152 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9nnq\" (UniqueName: \"kubernetes.io/projected/d5971011-f9d7-428a-951c-7e1f8484578a-kube-api-access-g9nnq\") pod \"maas-api-e2e-explicit-b838b7-8588bdf7cd-xmndp\" (UID: \"d5971011-f9d7-428a-951c-7e1f8484578a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b838b7-8588bdf7cd-xmndp" Sep 12 02:01:46.997360 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:46.997323 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5971011-f9d7-428a-951c-7e1f8484578a-maas-api-tls\") pod \"maas-api-e2e-explicit-b838b7-8588bdf7cd-xmndp\" (UID: \"d5971011-f9d7-428a-951c-7e1f8484578a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b838b7-8588bdf7cd-xmndp" Sep 12 02:01:47.037599 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:47.037540 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:01:47.098228 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:47.098188 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5971011-f9d7-428a-951c-7e1f8484578a-maas-api-tls\") pod \"maas-api-e2e-explicit-b838b7-8588bdf7cd-xmndp\" (UID: \"d5971011-f9d7-428a-951c-7e1f8484578a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b838b7-8588bdf7cd-xmndp" Sep 12 02:01:47.098404 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:47.098246 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5971011-f9d7-428a-951c-7e1f8484578a-openshift-service-ca\") pod \"maas-api-e2e-explicit-b838b7-8588bdf7cd-xmndp\" (UID: \"d5971011-f9d7-428a-951c-7e1f8484578a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b838b7-8588bdf7cd-xmndp" Sep 12 02:01:47.098404 ip-10-0-133-79 kubenswrapper[2579]: E0912 02:01:47.098363 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-b838b7-serving-cert: secret "maas-api-e2e-explicit-b838b7-serving-cert" not found Sep 12 02:01:47.098404 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:47.098373 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5971011-f9d7-428a-951c-7e1f8484578a-metrics-certs\") pod \"maas-api-e2e-explicit-b838b7-8588bdf7cd-xmndp\" (UID: \"d5971011-f9d7-428a-951c-7e1f8484578a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b838b7-8588bdf7cd-xmndp" Sep 12 02:01:47.098849 ip-10-0-133-79 kubenswrapper[2579]: E0912 02:01:47.098487 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5971011-f9d7-428a-951c-7e1f8484578a-maas-api-tls podName:d5971011-f9d7-428a-951c-7e1f8484578a nodeName:}" failed. No retries permitted until 2026-09-12 02:01:47.598428718 +0000 UTC m=+1010.390422492 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d5971011-f9d7-428a-951c-7e1f8484578a-maas-api-tls") pod "maas-api-e2e-explicit-b838b7-8588bdf7cd-xmndp" (UID: "d5971011-f9d7-428a-951c-7e1f8484578a") : secret "maas-api-e2e-explicit-b838b7-serving-cert" not found Sep 12 02:01:47.098849 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:47.098511 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9nnq\" (UniqueName: \"kubernetes.io/projected/d5971011-f9d7-428a-951c-7e1f8484578a-kube-api-access-g9nnq\") pod \"maas-api-e2e-explicit-b838b7-8588bdf7cd-xmndp\" (UID: \"d5971011-f9d7-428a-951c-7e1f8484578a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b838b7-8588bdf7cd-xmndp" Sep 12 02:01:47.099069 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:47.099040 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5971011-f9d7-428a-951c-7e1f8484578a-openshift-service-ca\") pod \"maas-api-e2e-explicit-b838b7-8588bdf7cd-xmndp\" (UID: \"d5971011-f9d7-428a-951c-7e1f8484578a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b838b7-8588bdf7cd-xmndp" Sep 12 02:01:47.101029 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:47.101014 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5971011-f9d7-428a-951c-7e1f8484578a-metrics-certs\") pod \"maas-api-e2e-explicit-b838b7-8588bdf7cd-xmndp\" (UID: \"d5971011-f9d7-428a-951c-7e1f8484578a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b838b7-8588bdf7cd-xmndp" Sep 12 02:01:47.121265 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:47.121232 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9nnq\" (UniqueName: \"kubernetes.io/projected/d5971011-f9d7-428a-951c-7e1f8484578a-kube-api-access-g9nnq\") pod \"maas-api-e2e-explicit-b838b7-8588bdf7cd-xmndp\" (UID: \"d5971011-f9d7-428a-951c-7e1f8484578a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b838b7-8588bdf7cd-xmndp" Sep 12 02:01:47.375745 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:47.375706 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:01:47.474005 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:47.473969 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:01:47.531528 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:47.531482 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b838b7-cc4d5586-z6vzt"] Sep 12 02:01:47.535253 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:47.535233 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b838b7-cc4d5586-z6vzt" Sep 12 02:01:47.538040 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:47.538016 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-b838b7-dockercfg-92dkj\"" Sep 12 02:01:47.538255 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:47.538240 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-b838b7\"" Sep 12 02:01:47.546738 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:47.546714 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b838b7-cc4d5586-z6vzt"] Sep 12 02:01:47.564649 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:47.564612 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:01:47.604215 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:47.604172 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqhs9\" (UniqueName: \"kubernetes.io/projected/5052265f-0402-428d-bf38-51f62c950021-kube-api-access-vqhs9\") pod \"payload-pre-processing-e2e-explicit-b838b7-cc4d5586-z6vzt\" (UID: \"5052265f-0402-428d-bf38-51f62c950021\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b838b7-cc4d5586-z6vzt" Sep 12 02:01:47.604215 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:47.604217 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5052265f-0402-428d-bf38-51f62c950021-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-b838b7-cc4d5586-z6vzt\" (UID: \"5052265f-0402-428d-bf38-51f62c950021\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b838b7-cc4d5586-z6vzt" Sep 12 02:01:47.604509 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:47.604344 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5971011-f9d7-428a-951c-7e1f8484578a-maas-api-tls\") pod \"maas-api-e2e-explicit-b838b7-8588bdf7cd-xmndp\" (UID: \"d5971011-f9d7-428a-951c-7e1f8484578a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b838b7-8588bdf7cd-xmndp" Sep 12 02:01:47.607146 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:47.607110 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5971011-f9d7-428a-951c-7e1f8484578a-maas-api-tls\") pod \"maas-api-e2e-explicit-b838b7-8588bdf7cd-xmndp\" (UID: \"d5971011-f9d7-428a-951c-7e1f8484578a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b838b7-8588bdf7cd-xmndp" Sep 12 02:01:47.705658 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:47.705562 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqhs9\" (UniqueName: \"kubernetes.io/projected/5052265f-0402-428d-bf38-51f62c950021-kube-api-access-vqhs9\") pod \"payload-pre-processing-e2e-explicit-b838b7-cc4d5586-z6vzt\" (UID: \"5052265f-0402-428d-bf38-51f62c950021\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b838b7-cc4d5586-z6vzt" Sep 12 02:01:47.705658 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:47.705606 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5052265f-0402-428d-bf38-51f62c950021-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-b838b7-cc4d5586-z6vzt\" (UID: \"5052265f-0402-428d-bf38-51f62c950021\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b838b7-cc4d5586-z6vzt" Sep 12 02:01:47.706303 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:47.706286 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5052265f-0402-428d-bf38-51f62c950021-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-b838b7-cc4d5586-z6vzt\" (UID: \"5052265f-0402-428d-bf38-51f62c950021\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b838b7-cc4d5586-z6vzt" Sep 12 02:01:47.714331 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:47.714295 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqhs9\" (UniqueName: \"kubernetes.io/projected/5052265f-0402-428d-bf38-51f62c950021-kube-api-access-vqhs9\") pod \"payload-pre-processing-e2e-explicit-b838b7-cc4d5586-z6vzt\" (UID: \"5052265f-0402-428d-bf38-51f62c950021\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b838b7-cc4d5586-z6vzt" Sep 12 02:01:47.845538 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:47.845491 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b838b7-8588bdf7cd-xmndp" Sep 12 02:01:47.847578 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:47.847556 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b838b7-cc4d5586-z6vzt" Sep 12 02:01:48.131232 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:48.131196 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b838b7-cc4d5586-z6vzt"] Sep 12 02:01:48.133937 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:48.133915 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b838b7-8588bdf7cd-xmndp"] Sep 12 02:01:48.134877 ip-10-0-133-79 kubenswrapper[2579]: W0912 02:01:48.134848 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5052265f_0402_428d_bf38_51f62c950021.slice/crio-232d5dcddfb8d0c4f0df9a594a996edba519781a668d87d629da4091443d9b48 WatchSource:0}: Error finding container 232d5dcddfb8d0c4f0df9a594a996edba519781a668d87d629da4091443d9b48: Status 404 returned error can't find the container with id 232d5dcddfb8d0c4f0df9a594a996edba519781a668d87d629da4091443d9b48 Sep 12 02:01:48.135991 ip-10-0-133-79 kubenswrapper[2579]: W0912 02:01:48.135958 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5971011_f9d7_428a_951c_7e1f8484578a.slice/crio-5c6c0f0f8909b69d7b617e610487b531076c27b75cb706d8e92fb0172ad13492 WatchSource:0}: Error finding container 5c6c0f0f8909b69d7b617e610487b531076c27b75cb706d8e92fb0172ad13492: Status 404 returned error can't find the container with id 5c6c0f0f8909b69d7b617e610487b531076c27b75cb706d8e92fb0172ad13492 Sep 12 02:01:48.137565 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:48.137547 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 12 02:01:48.823202 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:48.823165 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b838b7-8588bdf7cd-xmndp" event={"ID":"d5971011-f9d7-428a-951c-7e1f8484578a","Type":"ContainerStarted","Data":"686b4150190c3f265df98208b6bcaa4febe94ffdaa6d652aa45b6beb0da57747"} Sep 12 02:01:48.823689 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:48.823382 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b838b7-8588bdf7cd-xmndp" Sep 12 02:01:48.823689 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:48.823411 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b838b7-8588bdf7cd-xmndp" event={"ID":"d5971011-f9d7-428a-951c-7e1f8484578a","Type":"ContainerStarted","Data":"5c6c0f0f8909b69d7b617e610487b531076c27b75cb706d8e92fb0172ad13492"} Sep 12 02:01:48.824781 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:48.824753 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b838b7-cc4d5586-z6vzt" event={"ID":"5052265f-0402-428d-bf38-51f62c950021","Type":"ContainerStarted","Data":"e2d88c9089531bd0ccd6a798b52227ffc57b176bef5bd72e81e672adcc8e3634"} Sep 12 02:01:48.824897 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:48.824806 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b838b7-cc4d5586-z6vzt" Sep 12 02:01:48.824897 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:48.824829 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b838b7-cc4d5586-z6vzt" event={"ID":"5052265f-0402-428d-bf38-51f62c950021","Type":"ContainerStarted","Data":"232d5dcddfb8d0c4f0df9a594a996edba519781a668d87d629da4091443d9b48"} Sep 12 02:01:48.841971 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:48.841924 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b838b7-8588bdf7cd-xmndp" podStartSLOduration=2.311356722 podStartE2EDuration="2.841907788s" totalImagesPullingTime="530.551066ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:01:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:01:48.839281014 +0000 UTC m=+1011.631274810" watchObservedRunningTime="2026-09-12 02:01:48.841907788 +0000 UTC m=+1011.633901583" Sep 12 02:01:48.856159 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:48.856097 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b838b7-cc4d5586-z6vzt" podStartSLOduration=1.8560803049999999 podStartE2EDuration="1.856080305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:01:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:01:48.854165706 +0000 UTC m=+1011.646159500" watchObservedRunningTime="2026-09-12 02:01:48.856080305 +0000 UTC m=+1011.648074100" Sep 12 02:01:49.827592 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:49.827558 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:01:52.180995 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:52.180957 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:01:54.836178 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:54.836142 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b838b7-8588bdf7cd-xmndp" Sep 12 02:01:58.039160 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:58.039121 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:01:58.109229 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:58.109194 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:01:58.184120 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:58.184082 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:01:58.285650 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:58.285615 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:01:59.832798 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:01:59.832769 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b838b7-cc4d5586-z6vzt" Sep 12 02:02:04.624096 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:04.624066 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-7d6e9f/test-model-7d6e9f-kserve-8c9487947-5pxmt"] Sep 12 02:02:04.627984 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:04.627951 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7d6e9f/test-model-7d6e9f-kserve-8c9487947-5pxmt" Sep 12 02:02:04.630741 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:04.630719 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-7d6e9f\"/\"kube-root-ca.crt\"" Sep 12 02:02:04.630741 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:04.630737 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-7d6e9f\"/\"openshift-service-ca.crt\"" Sep 12 02:02:04.631573 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:04.631552 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-7d6e9f\"/\"test-model-7d6e9f-kserve-self-signed-certs\"" Sep 12 02:02:04.631686 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:04.631550 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-7d6e9f\"/\"default-dockercfg-cwr9p\"" Sep 12 02:02:04.640995 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:04.638835 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-7d6e9f/test-model-7d6e9f-kserve-8c9487947-5pxmt"] Sep 12 02:02:04.651252 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:04.651220 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:02:04.788806 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:04.788769 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a408541-bb13-47c3-a3ef-238f07e989fa-dshm\") pod \"test-model-7d6e9f-kserve-8c9487947-5pxmt\" (UID: \"6a408541-bb13-47c3-a3ef-238f07e989fa\") " pod="ai-tenant-e2e-inf-b-7d6e9f/test-model-7d6e9f-kserve-8c9487947-5pxmt" Sep 12 02:02:04.788985 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:04.788825 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a408541-bb13-47c3-a3ef-238f07e989fa-model-cache\") pod \"test-model-7d6e9f-kserve-8c9487947-5pxmt\" (UID: \"6a408541-bb13-47c3-a3ef-238f07e989fa\") " pod="ai-tenant-e2e-inf-b-7d6e9f/test-model-7d6e9f-kserve-8c9487947-5pxmt" Sep 12 02:02:04.788985 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:04.788902 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a408541-bb13-47c3-a3ef-238f07e989fa-tls-certs\") pod \"test-model-7d6e9f-kserve-8c9487947-5pxmt\" (UID: \"6a408541-bb13-47c3-a3ef-238f07e989fa\") " pod="ai-tenant-e2e-inf-b-7d6e9f/test-model-7d6e9f-kserve-8c9487947-5pxmt" Sep 12 02:02:04.788985 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:04.788935 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv9lk\" (UniqueName: \"kubernetes.io/projected/6a408541-bb13-47c3-a3ef-238f07e989fa-kube-api-access-mv9lk\") pod \"test-model-7d6e9f-kserve-8c9487947-5pxmt\" (UID: \"6a408541-bb13-47c3-a3ef-238f07e989fa\") " pod="ai-tenant-e2e-inf-b-7d6e9f/test-model-7d6e9f-kserve-8c9487947-5pxmt" Sep 12 02:02:04.788985 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:04.788961 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a408541-bb13-47c3-a3ef-238f07e989fa-tmp-dir\") pod \"test-model-7d6e9f-kserve-8c9487947-5pxmt\" (UID: \"6a408541-bb13-47c3-a3ef-238f07e989fa\") " pod="ai-tenant-e2e-inf-b-7d6e9f/test-model-7d6e9f-kserve-8c9487947-5pxmt" Sep 12 02:02:04.789121 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:04.788992 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a408541-bb13-47c3-a3ef-238f07e989fa-home\") pod \"test-model-7d6e9f-kserve-8c9487947-5pxmt\" (UID: \"6a408541-bb13-47c3-a3ef-238f07e989fa\") " pod="ai-tenant-e2e-inf-b-7d6e9f/test-model-7d6e9f-kserve-8c9487947-5pxmt" Sep 12 02:02:04.890273 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:04.890178 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a408541-bb13-47c3-a3ef-238f07e989fa-model-cache\") pod \"test-model-7d6e9f-kserve-8c9487947-5pxmt\" (UID: \"6a408541-bb13-47c3-a3ef-238f07e989fa\") " pod="ai-tenant-e2e-inf-b-7d6e9f/test-model-7d6e9f-kserve-8c9487947-5pxmt" Sep 12 02:02:04.890273 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:04.890251 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a408541-bb13-47c3-a3ef-238f07e989fa-tls-certs\") pod \"test-model-7d6e9f-kserve-8c9487947-5pxmt\" (UID: \"6a408541-bb13-47c3-a3ef-238f07e989fa\") " pod="ai-tenant-e2e-inf-b-7d6e9f/test-model-7d6e9f-kserve-8c9487947-5pxmt" Sep 12 02:02:04.890529 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:04.890273 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv9lk\" (UniqueName: \"kubernetes.io/projected/6a408541-bb13-47c3-a3ef-238f07e989fa-kube-api-access-mv9lk\") pod \"test-model-7d6e9f-kserve-8c9487947-5pxmt\" (UID: \"6a408541-bb13-47c3-a3ef-238f07e989fa\") " pod="ai-tenant-e2e-inf-b-7d6e9f/test-model-7d6e9f-kserve-8c9487947-5pxmt" Sep 12 02:02:04.890529 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:04.890302 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a408541-bb13-47c3-a3ef-238f07e989fa-tmp-dir\") pod \"test-model-7d6e9f-kserve-8c9487947-5pxmt\" (UID: \"6a408541-bb13-47c3-a3ef-238f07e989fa\") " pod="ai-tenant-e2e-inf-b-7d6e9f/test-model-7d6e9f-kserve-8c9487947-5pxmt" Sep 12 02:02:04.890674 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:04.890584 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a408541-bb13-47c3-a3ef-238f07e989fa-home\") pod \"test-model-7d6e9f-kserve-8c9487947-5pxmt\" (UID: \"6a408541-bb13-47c3-a3ef-238f07e989fa\") " pod="ai-tenant-e2e-inf-b-7d6e9f/test-model-7d6e9f-kserve-8c9487947-5pxmt" Sep 12 02:02:04.890747 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:04.890682 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a408541-bb13-47c3-a3ef-238f07e989fa-model-cache\") pod \"test-model-7d6e9f-kserve-8c9487947-5pxmt\" (UID: \"6a408541-bb13-47c3-a3ef-238f07e989fa\") " pod="ai-tenant-e2e-inf-b-7d6e9f/test-model-7d6e9f-kserve-8c9487947-5pxmt" Sep 12 02:02:04.890747 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:04.890703 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a408541-bb13-47c3-a3ef-238f07e989fa-tmp-dir\") pod \"test-model-7d6e9f-kserve-8c9487947-5pxmt\" (UID: \"6a408541-bb13-47c3-a3ef-238f07e989fa\") " pod="ai-tenant-e2e-inf-b-7d6e9f/test-model-7d6e9f-kserve-8c9487947-5pxmt" Sep 12 02:02:04.890747 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:04.890716 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a408541-bb13-47c3-a3ef-238f07e989fa-dshm\") pod \"test-model-7d6e9f-kserve-8c9487947-5pxmt\" (UID: \"6a408541-bb13-47c3-a3ef-238f07e989fa\") " pod="ai-tenant-e2e-inf-b-7d6e9f/test-model-7d6e9f-kserve-8c9487947-5pxmt" Sep 12 02:02:04.890915 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:04.890851 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a408541-bb13-47c3-a3ef-238f07e989fa-home\") pod \"test-model-7d6e9f-kserve-8c9487947-5pxmt\" (UID: \"6a408541-bb13-47c3-a3ef-238f07e989fa\") " pod="ai-tenant-e2e-inf-b-7d6e9f/test-model-7d6e9f-kserve-8c9487947-5pxmt" Sep 12 02:02:04.892858 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:04.892836 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a408541-bb13-47c3-a3ef-238f07e989fa-dshm\") pod \"test-model-7d6e9f-kserve-8c9487947-5pxmt\" (UID: \"6a408541-bb13-47c3-a3ef-238f07e989fa\") " pod="ai-tenant-e2e-inf-b-7d6e9f/test-model-7d6e9f-kserve-8c9487947-5pxmt" Sep 12 02:02:04.893091 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:04.893070 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a408541-bb13-47c3-a3ef-238f07e989fa-tls-certs\") pod \"test-model-7d6e9f-kserve-8c9487947-5pxmt\" (UID: \"6a408541-bb13-47c3-a3ef-238f07e989fa\") " pod="ai-tenant-e2e-inf-b-7d6e9f/test-model-7d6e9f-kserve-8c9487947-5pxmt" Sep 12 02:02:04.899919 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:04.899901 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv9lk\" (UniqueName: \"kubernetes.io/projected/6a408541-bb13-47c3-a3ef-238f07e989fa-kube-api-access-mv9lk\") pod \"test-model-7d6e9f-kserve-8c9487947-5pxmt\" (UID: \"6a408541-bb13-47c3-a3ef-238f07e989fa\") " pod="ai-tenant-e2e-inf-b-7d6e9f/test-model-7d6e9f-kserve-8c9487947-5pxmt" Sep 12 02:02:04.942651 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:04.942607 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7d6e9f/test-model-7d6e9f-kserve-8c9487947-5pxmt" Sep 12 02:02:05.215907 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:05.215881 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-7d6e9f/test-model-7d6e9f-kserve-8c9487947-5pxmt"] Sep 12 02:02:05.217836 ip-10-0-133-79 kubenswrapper[2579]: W0912 02:02:05.217796 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a408541_bb13_47c3_a3ef_238f07e989fa.slice/crio-0a996799430c1400d4b708900218ab15e737c2d55cd7af99af40784b934d8717 WatchSource:0}: Error finding container 0a996799430c1400d4b708900218ab15e737c2d55cd7af99af40784b934d8717: Status 404 returned error can't find the container with id 0a996799430c1400d4b708900218ab15e737c2d55cd7af99af40784b934d8717 Sep 12 02:02:05.919735 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:05.919699 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7d6e9f/test-model-7d6e9f-kserve-8c9487947-5pxmt" event={"ID":"6a408541-bb13-47c3-a3ef-238f07e989fa","Type":"ContainerStarted","Data":"bd85807b4ee9ce079a0c5a72f586b810e42369e1969b51c77930ecf68c96f636"} Sep 12 02:02:05.920113 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:05.919743 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7d6e9f/test-model-7d6e9f-kserve-8c9487947-5pxmt" event={"ID":"6a408541-bb13-47c3-a3ef-238f07e989fa","Type":"ContainerStarted","Data":"0a996799430c1400d4b708900218ab15e737c2d55cd7af99af40784b934d8717"} Sep 12 02:02:05.936879 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:05.936827 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-7d6e9f/test-model-7d6e9f-kserve-8c9487947-5pxmt" podStartSLOduration=1.936810412 podStartE2EDuration="1.936810412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:02:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:02:05.9351697 +0000 UTC m=+1028.727163510" watchObservedRunningTime="2026-09-12 02:02:05.936810412 +0000 UTC m=+1028.728804207" Sep 12 02:02:06.822177 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:06.822144 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:02:08.761850 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:08.761811 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:02:09.643430 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:09.643395 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:02:09.758828 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:09.758789 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:02:09.860163 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:09.860119 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:02:10.589637 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:10.589600 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:02:11.234918 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:11.234880 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:02:11.885473 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:11.885424 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:02:12.529214 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:12.529176 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:02:13.331855 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:13.331822 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:02:13.909257 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:13.909214 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:02:14.943198 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:14.943166 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-7d6e9f/test-model-7d6e9f-kserve-8c9487947-5pxmt" Sep 12 02:02:14.943639 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:14.943220 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-7d6e9f/test-model-7d6e9f-kserve-8c9487947-5pxmt" Sep 12 02:02:14.956347 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:14.956320 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-7d6e9f/test-model-7d6e9f-kserve-8c9487947-5pxmt" Sep 12 02:02:14.982481 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:14.979510 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-7d6e9f/test-model-7d6e9f-kserve-8c9487947-5pxmt" Sep 12 02:02:15.656615 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:15.656572 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:02:15.762419 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:15.762384 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:02:16.007365 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:16.007272 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:02:16.107436 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:16.107400 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:02:16.653461 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:16.653411 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:02:16.750416 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:16.750381 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:02:16.852610 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:16.852575 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:02:18.443041 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:18.443003 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:02:24.139668 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:24.139631 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:02:24.239839 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:24.239798 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:02:24.842907 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:24.842866 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:02:26.234216 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:26.234186 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:02:30.578886 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:30.578849 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:02:30.680060 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:30.680018 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:02:30.816682 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:30.816655 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:02:31.144911 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:31.144874 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:02:31.247861 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:31.247824 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:02:36.243717 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:36.243675 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:02:36.880840 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:36.880805 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:02:37.032287 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:37.032253 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:02:37.132407 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:37.132312 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:02:37.771741 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:37.771705 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:02:38.486037 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:38.486000 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:02:39.157289 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:39.157252 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:02:39.890545 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:39.890506 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:02:40.650146 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:40.650090 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:02:40.751110 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:40.751058 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:02:40.899423 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:40.899389 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:02:41.104012 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:41.103924 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:02:41.206975 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:41.206928 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:02:41.217018 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:41.216985 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d5764ff8-7b87c99754-8ndt2"] Sep 12 02:02:41.220622 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:41.220606 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d5764ff8-7b87c99754-8ndt2" Sep 12 02:02:41.222978 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:41.222957 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-d5764ff8-dockercfg-xswrt\"" Sep 12 02:02:41.223097 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:41.222963 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-d5764ff8\"" Sep 12 02:02:41.230053 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:41.230030 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d5764ff8-7b87c99754-8ndt2"] Sep 12 02:02:41.252692 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:41.252655 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d5764ff8-747cdc84b5-vlcbm"] Sep 12 02:02:41.257153 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:41.257133 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d5764ff8-747cdc84b5-vlcbm" Sep 12 02:02:41.264184 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:41.264144 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d5764ff8-747cdc84b5-vlcbm"] Sep 12 02:02:41.357578 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:41.357501 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eed52bc-6546-4ed3-9c97-fd34057897a7-plugins-config-volume\") pod \"payload-processing-e2e-mt-d5764ff8-747cdc84b5-vlcbm\" (UID: \"6eed52bc-6546-4ed3-9c97-fd34057897a7\") " pod="openshift-ingress/payload-processing-e2e-mt-d5764ff8-747cdc84b5-vlcbm" Sep 12 02:02:41.357578 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:41.357543 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrgr4\" (UniqueName: \"kubernetes.io/projected/6eed52bc-6546-4ed3-9c97-fd34057897a7-kube-api-access-lrgr4\") pod \"payload-processing-e2e-mt-d5764ff8-747cdc84b5-vlcbm\" (UID: \"6eed52bc-6546-4ed3-9c97-fd34057897a7\") " pod="openshift-ingress/payload-processing-e2e-mt-d5764ff8-747cdc84b5-vlcbm" Sep 12 02:02:41.357760 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:41.357652 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8f7g\" (UniqueName: \"kubernetes.io/projected/7697cc26-c2ff-46d8-bc9b-2ac5a72328ba-kube-api-access-m8f7g\") pod \"payload-pre-processing-e2e-mt-d5764ff8-7b87c99754-8ndt2\" (UID: \"7697cc26-c2ff-46d8-bc9b-2ac5a72328ba\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d5764ff8-7b87c99754-8ndt2" Sep 12 02:02:41.357760 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:41.357686 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7697cc26-c2ff-46d8-bc9b-2ac5a72328ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d5764ff8-7b87c99754-8ndt2\" (UID: \"7697cc26-c2ff-46d8-bc9b-2ac5a72328ba\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d5764ff8-7b87c99754-8ndt2" Sep 12 02:02:41.458690 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:41.458656 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eed52bc-6546-4ed3-9c97-fd34057897a7-plugins-config-volume\") pod \"payload-processing-e2e-mt-d5764ff8-747cdc84b5-vlcbm\" (UID: \"6eed52bc-6546-4ed3-9c97-fd34057897a7\") " pod="openshift-ingress/payload-processing-e2e-mt-d5764ff8-747cdc84b5-vlcbm" Sep 12 02:02:41.458690 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:41.458696 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrgr4\" (UniqueName: \"kubernetes.io/projected/6eed52bc-6546-4ed3-9c97-fd34057897a7-kube-api-access-lrgr4\") pod \"payload-processing-e2e-mt-d5764ff8-747cdc84b5-vlcbm\" (UID: \"6eed52bc-6546-4ed3-9c97-fd34057897a7\") " pod="openshift-ingress/payload-processing-e2e-mt-d5764ff8-747cdc84b5-vlcbm" Sep 12 02:02:41.458899 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:41.458753 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8f7g\" (UniqueName: \"kubernetes.io/projected/7697cc26-c2ff-46d8-bc9b-2ac5a72328ba-kube-api-access-m8f7g\") pod \"payload-pre-processing-e2e-mt-d5764ff8-7b87c99754-8ndt2\" (UID: \"7697cc26-c2ff-46d8-bc9b-2ac5a72328ba\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d5764ff8-7b87c99754-8ndt2" Sep 12 02:02:41.458899 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:41.458779 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7697cc26-c2ff-46d8-bc9b-2ac5a72328ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d5764ff8-7b87c99754-8ndt2\" (UID: \"7697cc26-c2ff-46d8-bc9b-2ac5a72328ba\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d5764ff8-7b87c99754-8ndt2" Sep 12 02:02:41.459410 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:41.459391 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eed52bc-6546-4ed3-9c97-fd34057897a7-plugins-config-volume\") pod \"payload-processing-e2e-mt-d5764ff8-747cdc84b5-vlcbm\" (UID: \"6eed52bc-6546-4ed3-9c97-fd34057897a7\") " pod="openshift-ingress/payload-processing-e2e-mt-d5764ff8-747cdc84b5-vlcbm" Sep 12 02:02:41.459486 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:41.459425 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7697cc26-c2ff-46d8-bc9b-2ac5a72328ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d5764ff8-7b87c99754-8ndt2\" (UID: \"7697cc26-c2ff-46d8-bc9b-2ac5a72328ba\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d5764ff8-7b87c99754-8ndt2" Sep 12 02:02:41.467794 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:41.467772 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrgr4\" (UniqueName: \"kubernetes.io/projected/6eed52bc-6546-4ed3-9c97-fd34057897a7-kube-api-access-lrgr4\") pod \"payload-processing-e2e-mt-d5764ff8-747cdc84b5-vlcbm\" (UID: \"6eed52bc-6546-4ed3-9c97-fd34057897a7\") " pod="openshift-ingress/payload-processing-e2e-mt-d5764ff8-747cdc84b5-vlcbm" Sep 12 02:02:41.467953 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:41.467933 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8f7g\" (UniqueName: \"kubernetes.io/projected/7697cc26-c2ff-46d8-bc9b-2ac5a72328ba-kube-api-access-m8f7g\") pod \"payload-pre-processing-e2e-mt-d5764ff8-7b87c99754-8ndt2\" (UID: \"7697cc26-c2ff-46d8-bc9b-2ac5a72328ba\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d5764ff8-7b87c99754-8ndt2" Sep 12 02:02:41.533146 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:41.533117 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d5764ff8-7b87c99754-8ndt2" Sep 12 02:02:41.570474 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:41.570422 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d5764ff8-747cdc84b5-vlcbm" Sep 12 02:02:41.811163 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:41.811139 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d5764ff8-7b87c99754-8ndt2"] Sep 12 02:02:41.813218 ip-10-0-133-79 kubenswrapper[2579]: W0912 02:02:41.813192 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7697cc26_c2ff_46d8_bc9b_2ac5a72328ba.slice/crio-7dca4f904aa627b7b85656f485c4c48758ba9936ac89d6dc1e240648ce8b04c2 WatchSource:0}: Error finding container 7dca4f904aa627b7b85656f485c4c48758ba9936ac89d6dc1e240648ce8b04c2: Status 404 returned error can't find the container with id 7dca4f904aa627b7b85656f485c4c48758ba9936ac89d6dc1e240648ce8b04c2 Sep 12 02:02:41.867273 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:41.867249 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d5764ff8-747cdc84b5-vlcbm"] Sep 12 02:02:41.869543 ip-10-0-133-79 kubenswrapper[2579]: W0912 02:02:41.869506 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eed52bc_6546_4ed3_9c97_fd34057897a7.slice/crio-7b51afd2ea4a9bdec3566223a0ff36cff96f696476968206a503658b5d4e3990 WatchSource:0}: Error finding container 7b51afd2ea4a9bdec3566223a0ff36cff96f696476968206a503658b5d4e3990: Status 404 returned error can't find the container with id 7b51afd2ea4a9bdec3566223a0ff36cff96f696476968206a503658b5d4e3990 Sep 12 02:02:42.116562 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:42.116512 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d5764ff8-747cdc84b5-vlcbm" event={"ID":"6eed52bc-6546-4ed3-9c97-fd34057897a7","Type":"ContainerStarted","Data":"859692dfad20e815545a1db0d9cb6199c432f4876e3ed7406de9273f9fe51280"} Sep 12 02:02:42.116761 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:42.116628 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-d5764ff8-747cdc84b5-vlcbm" Sep 12 02:02:42.116761 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:42.116655 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d5764ff8-747cdc84b5-vlcbm" event={"ID":"6eed52bc-6546-4ed3-9c97-fd34057897a7","Type":"ContainerStarted","Data":"7b51afd2ea4a9bdec3566223a0ff36cff96f696476968206a503658b5d4e3990"} Sep 12 02:02:42.118490 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:42.118434 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d5764ff8-7b87c99754-8ndt2" event={"ID":"7697cc26-c2ff-46d8-bc9b-2ac5a72328ba","Type":"ContainerStarted","Data":"cfb0d6f36636561e5e18e1fd8cf969b61132a8a4daf7653c54357af6cbe3f498"} Sep 12 02:02:42.118676 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:42.118518 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d5764ff8-7b87c99754-8ndt2" Sep 12 02:02:42.118676 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:42.118542 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d5764ff8-7b87c99754-8ndt2" event={"ID":"7697cc26-c2ff-46d8-bc9b-2ac5a72328ba","Type":"ContainerStarted","Data":"7dca4f904aa627b7b85656f485c4c48758ba9936ac89d6dc1e240648ce8b04c2"} Sep 12 02:02:42.134425 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:42.134367 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-d5764ff8-747cdc84b5-vlcbm" podStartSLOduration=1.134350295 podStartE2EDuration="1.134350295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:02:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:02:42.133076284 +0000 UTC m=+1064.925070080" watchObservedRunningTime="2026-09-12 02:02:42.134350295 +0000 UTC m=+1064.926344089" Sep 12 02:02:42.149552 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:42.149437 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-d5764ff8-7b87c99754-8ndt2" podStartSLOduration=1.149419633 podStartE2EDuration="1.149419633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:02:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:02:42.147918871 +0000 UTC m=+1064.939912669" watchObservedRunningTime="2026-09-12 02:02:42.149419633 +0000 UTC m=+1064.941413425" Sep 12 02:02:51.654039 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:51.653958 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:02:53.134328 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:53.134293 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-d5764ff8-747cdc84b5-vlcbm" Sep 12 02:02:53.134796 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:53.134390 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d5764ff8-7b87c99754-8ndt2" Sep 12 02:02:55.033741 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:02:55.033707 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:03:01.060281 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:01.060244 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:02.650782 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:02.650741 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:03.427339 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:03.427306 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:04.067769 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:04.067735 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:04.851667 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:04.851631 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:05.680388 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:05.680360 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:06.464878 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:06.464834 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:07.207504 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:07.207464 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:07.308954 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:07.308916 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:07.902120 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:07.902088 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:08.002497 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:08.002463 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:08.359346 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:08.359255 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:09.309038 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:09.308997 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:09.369232 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:09.369199 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:10.963590 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:10.963558 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:13.933023 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:13.932977 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:13.992969 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:13.992933 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:14.372950 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:14.372918 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:15.047294 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:15.047247 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:15.792999 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:15.792959 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:16.684760 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:16.684709 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:17.257986 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:17.257951 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:18.155968 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:18.155935 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:18.840766 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:18.840731 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:19.551926 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:19.551883 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:20.044388 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:20.044349 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:24.700791 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:24.700750 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b838b7-8588bdf7cd-xmndp"] Sep 12 02:03:24.701675 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:24.701643 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b838b7-8588bdf7cd-xmndp" podUID="d5971011-f9d7-428a-951c-7e1f8484578a" containerName="maas-api" containerID="cri-o://686b4150190c3f265df98208b6bcaa4febe94ffdaa6d652aa45b6beb0da57747" gracePeriod=30 Sep 12 02:03:24.729361 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:24.729320 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:24.759592 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:24.759554 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b838b7-cc4d5586-z6vzt"] Sep 12 02:03:24.759852 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:24.759826 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b838b7-cc4d5586-z6vzt" podUID="5052265f-0402-428d-bf38-51f62c950021" containerName="payload-pre-processing" containerID="cri-o://e2d88c9089531bd0ccd6a798b52227ffc57b176bef5bd72e81e672adcc8e3634" gracePeriod=30 Sep 12 02:03:24.802373 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:24.801883 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:24.831285 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:24.831216 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b838b7-8588bdf7cd-xmndp" podUID="d5971011-f9d7-428a-951c-7e1f8484578a" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.54:8443/health\": dial tcp 10.133.0.54:8443: connect: connection refused" Sep 12 02:03:24.904072 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:24.903551 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:24.904376 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:24.904281 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d5764ff8-747cdc84b5-vlcbm"] Sep 12 02:03:24.905870 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:24.905620 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-d5764ff8-747cdc84b5-vlcbm" podUID="6eed52bc-6546-4ed3-9c97-fd34057897a7" containerName="payload-processing" containerID="cri-o://859692dfad20e815545a1db0d9cb6199c432f4876e3ed7406de9273f9fe51280" gracePeriod=30 Sep 12 02:03:24.909937 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:24.909810 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d5764ff8-7b87c99754-8ndt2"] Sep 12 02:03:24.910069 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:24.910030 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-d5764ff8-7b87c99754-8ndt2" podUID="7697cc26-c2ff-46d8-bc9b-2ac5a72328ba" containerName="payload-pre-processing" containerID="cri-o://cfb0d6f36636561e5e18e1fd8cf969b61132a8a4daf7653c54357af6cbe3f498" gracePeriod=30 Sep 12 02:03:24.999201 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:24.999165 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:25.059362 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.059202 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:25.076850 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.076825 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b838b7-8588bdf7cd-xmndp" Sep 12 02:03:25.079759 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.079736 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b838b7-cc4d5586-z6vzt" Sep 12 02:03:25.104827 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.104798 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vqhs9\" (UniqueName: \"kubernetes.io/projected/5052265f-0402-428d-bf38-51f62c950021-kube-api-access-vqhs9\") pod \"5052265f-0402-428d-bf38-51f62c950021\" (UID: \"5052265f-0402-428d-bf38-51f62c950021\") " Sep 12 02:03:25.104990 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.104895 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5971011-f9d7-428a-951c-7e1f8484578a-metrics-certs\") pod \"d5971011-f9d7-428a-951c-7e1f8484578a\" (UID: \"d5971011-f9d7-428a-951c-7e1f8484578a\") " Sep 12 02:03:25.104990 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.104947 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5971011-f9d7-428a-951c-7e1f8484578a-maas-api-tls\") pod \"d5971011-f9d7-428a-951c-7e1f8484578a\" (UID: \"d5971011-f9d7-428a-951c-7e1f8484578a\") " Sep 12 02:03:25.105254 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.104992 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9nnq\" (UniqueName: \"kubernetes.io/projected/d5971011-f9d7-428a-951c-7e1f8484578a-kube-api-access-g9nnq\") pod \"d5971011-f9d7-428a-951c-7e1f8484578a\" (UID: \"d5971011-f9d7-428a-951c-7e1f8484578a\") " Sep 12 02:03:25.105254 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.105046 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5971011-f9d7-428a-951c-7e1f8484578a-openshift-service-ca\") pod \"d5971011-f9d7-428a-951c-7e1f8484578a\" (UID: \"d5971011-f9d7-428a-951c-7e1f8484578a\") " Sep 12 02:03:25.105377 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.105282 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5052265f-0402-428d-bf38-51f62c950021-plugins-config-volume\") pod \"5052265f-0402-428d-bf38-51f62c950021\" (UID: \"5052265f-0402-428d-bf38-51f62c950021\") " Sep 12 02:03:25.105944 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.105681 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5971011-f9d7-428a-951c-7e1f8484578a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d5971011-f9d7-428a-951c-7e1f8484578a" (UID: "d5971011-f9d7-428a-951c-7e1f8484578a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 12 02:03:25.105944 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.105702 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5052265f-0402-428d-bf38-51f62c950021-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5052265f-0402-428d-bf38-51f62c950021" (UID: "5052265f-0402-428d-bf38-51f62c950021"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 12 02:03:25.105944 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.105805 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5971011-f9d7-428a-951c-7e1f8484578a-openshift-service-ca\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:03:25.105944 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.105827 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5052265f-0402-428d-bf38-51f62c950021-plugins-config-volume\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:03:25.108568 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.108530 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5971011-f9d7-428a-951c-7e1f8484578a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d5971011-f9d7-428a-951c-7e1f8484578a" (UID: "d5971011-f9d7-428a-951c-7e1f8484578a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 12 02:03:25.108568 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.108543 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5971011-f9d7-428a-951c-7e1f8484578a-kube-api-access-g9nnq" (OuterVolumeSpecName: "kube-api-access-g9nnq") pod "d5971011-f9d7-428a-951c-7e1f8484578a" (UID: "d5971011-f9d7-428a-951c-7e1f8484578a"). InnerVolumeSpecName "kube-api-access-g9nnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 12 02:03:25.108711 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.108623 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5052265f-0402-428d-bf38-51f62c950021-kube-api-access-vqhs9" (OuterVolumeSpecName: "kube-api-access-vqhs9") pod "5052265f-0402-428d-bf38-51f62c950021" (UID: "5052265f-0402-428d-bf38-51f62c950021"). InnerVolumeSpecName "kube-api-access-vqhs9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 12 02:03:25.108711 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.108670 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5971011-f9d7-428a-951c-7e1f8484578a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d5971011-f9d7-428a-951c-7e1f8484578a" (UID: "d5971011-f9d7-428a-951c-7e1f8484578a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 12 02:03:25.180156 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.180131 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d5764ff8-747cdc84b5-vlcbm" Sep 12 02:03:25.194805 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.194783 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d5764ff8-7b87c99754-8ndt2" Sep 12 02:03:25.206720 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.206694 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrgr4\" (UniqueName: \"kubernetes.io/projected/6eed52bc-6546-4ed3-9c97-fd34057897a7-kube-api-access-lrgr4\") pod \"6eed52bc-6546-4ed3-9c97-fd34057897a7\" (UID: \"6eed52bc-6546-4ed3-9c97-fd34057897a7\") " Sep 12 02:03:25.206896 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.206781 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eed52bc-6546-4ed3-9c97-fd34057897a7-plugins-config-volume\") pod \"6eed52bc-6546-4ed3-9c97-fd34057897a7\" (UID: \"6eed52bc-6546-4ed3-9c97-fd34057897a7\") " Sep 12 02:03:25.207058 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.207041 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9nnq\" (UniqueName: \"kubernetes.io/projected/d5971011-f9d7-428a-951c-7e1f8484578a-kube-api-access-g9nnq\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:03:25.207123 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.207061 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqhs9\" (UniqueName: \"kubernetes.io/projected/5052265f-0402-428d-bf38-51f62c950021-kube-api-access-vqhs9\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:03:25.207123 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.207076 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5971011-f9d7-428a-951c-7e1f8484578a-metrics-certs\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:03:25.207123 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.207084 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5971011-f9d7-428a-951c-7e1f8484578a-maas-api-tls\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:03:25.207226 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.207113 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6eed52bc-6546-4ed3-9c97-fd34057897a7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6eed52bc-6546-4ed3-9c97-fd34057897a7" (UID: "6eed52bc-6546-4ed3-9c97-fd34057897a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 12 02:03:25.208921 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.208898 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eed52bc-6546-4ed3-9c97-fd34057897a7-kube-api-access-lrgr4" (OuterVolumeSpecName: "kube-api-access-lrgr4") pod "6eed52bc-6546-4ed3-9c97-fd34057897a7" (UID: "6eed52bc-6546-4ed3-9c97-fd34057897a7"). InnerVolumeSpecName "kube-api-access-lrgr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 12 02:03:25.230551 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.230469 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:25.307512 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.307467 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7697cc26-c2ff-46d8-bc9b-2ac5a72328ba-plugins-config-volume\") pod \"7697cc26-c2ff-46d8-bc9b-2ac5a72328ba\" (UID: \"7697cc26-c2ff-46d8-bc9b-2ac5a72328ba\") " Sep 12 02:03:25.307512 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.307518 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8f7g\" (UniqueName: \"kubernetes.io/projected/7697cc26-c2ff-46d8-bc9b-2ac5a72328ba-kube-api-access-m8f7g\") pod \"7697cc26-c2ff-46d8-bc9b-2ac5a72328ba\" (UID: \"7697cc26-c2ff-46d8-bc9b-2ac5a72328ba\") " Sep 12 02:03:25.307857 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.307833 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7697cc26-c2ff-46d8-bc9b-2ac5a72328ba-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7697cc26-c2ff-46d8-bc9b-2ac5a72328ba" (UID: "7697cc26-c2ff-46d8-bc9b-2ac5a72328ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 12 02:03:25.307932 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.307859 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrgr4\" (UniqueName: \"kubernetes.io/projected/6eed52bc-6546-4ed3-9c97-fd34057897a7-kube-api-access-lrgr4\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:03:25.307932 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.307879 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eed52bc-6546-4ed3-9c97-fd34057897a7-plugins-config-volume\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:03:25.309740 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.309720 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7697cc26-c2ff-46d8-bc9b-2ac5a72328ba-kube-api-access-m8f7g" (OuterVolumeSpecName: "kube-api-access-m8f7g") pod "7697cc26-c2ff-46d8-bc9b-2ac5a72328ba" (UID: "7697cc26-c2ff-46d8-bc9b-2ac5a72328ba"). InnerVolumeSpecName "kube-api-access-m8f7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 12 02:03:25.370075 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.370044 2579 generic.go:356] "Generic (PLEG): container finished" podID="5052265f-0402-428d-bf38-51f62c950021" containerID="e2d88c9089531bd0ccd6a798b52227ffc57b176bef5bd72e81e672adcc8e3634" exitCode=0 Sep 12 02:03:25.370269 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.370101 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b838b7-cc4d5586-z6vzt" Sep 12 02:03:25.370269 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.370147 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b838b7-cc4d5586-z6vzt" event={"ID":"5052265f-0402-428d-bf38-51f62c950021","Type":"ContainerDied","Data":"e2d88c9089531bd0ccd6a798b52227ffc57b176bef5bd72e81e672adcc8e3634"} Sep 12 02:03:25.370269 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.370196 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b838b7-cc4d5586-z6vzt" event={"ID":"5052265f-0402-428d-bf38-51f62c950021","Type":"ContainerDied","Data":"232d5dcddfb8d0c4f0df9a594a996edba519781a668d87d629da4091443d9b48"} Sep 12 02:03:25.370269 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.370226 2579 scope.go:117] "RemoveContainer" containerID="e2d88c9089531bd0ccd6a798b52227ffc57b176bef5bd72e81e672adcc8e3634" Sep 12 02:03:25.371585 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.371566 2579 generic.go:356] "Generic (PLEG): container finished" podID="d5971011-f9d7-428a-951c-7e1f8484578a" containerID="686b4150190c3f265df98208b6bcaa4febe94ffdaa6d652aa45b6beb0da57747" exitCode=0 Sep 12 02:03:25.371680 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.371616 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b838b7-8588bdf7cd-xmndp" Sep 12 02:03:25.371748 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.371679 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b838b7-8588bdf7cd-xmndp" event={"ID":"d5971011-f9d7-428a-951c-7e1f8484578a","Type":"ContainerDied","Data":"686b4150190c3f265df98208b6bcaa4febe94ffdaa6d652aa45b6beb0da57747"} Sep 12 02:03:25.371807 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.371750 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b838b7-8588bdf7cd-xmndp" event={"ID":"d5971011-f9d7-428a-951c-7e1f8484578a","Type":"ContainerDied","Data":"5c6c0f0f8909b69d7b617e610487b531076c27b75cb706d8e92fb0172ad13492"} Sep 12 02:03:25.373545 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.373524 2579 generic.go:356] "Generic (PLEG): container finished" podID="6eed52bc-6546-4ed3-9c97-fd34057897a7" containerID="859692dfad20e815545a1db0d9cb6199c432f4876e3ed7406de9273f9fe51280" exitCode=0 Sep 12 02:03:25.373695 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.373574 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d5764ff8-747cdc84b5-vlcbm" Sep 12 02:03:25.373695 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.373599 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d5764ff8-747cdc84b5-vlcbm" event={"ID":"6eed52bc-6546-4ed3-9c97-fd34057897a7","Type":"ContainerDied","Data":"859692dfad20e815545a1db0d9cb6199c432f4876e3ed7406de9273f9fe51280"} Sep 12 02:03:25.373695 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.373650 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d5764ff8-747cdc84b5-vlcbm" event={"ID":"6eed52bc-6546-4ed3-9c97-fd34057897a7","Type":"ContainerDied","Data":"7b51afd2ea4a9bdec3566223a0ff36cff96f696476968206a503658b5d4e3990"} Sep 12 02:03:25.374991 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.374969 2579 generic.go:356] "Generic (PLEG): container finished" podID="7697cc26-c2ff-46d8-bc9b-2ac5a72328ba" containerID="cfb0d6f36636561e5e18e1fd8cf969b61132a8a4daf7653c54357af6cbe3f498" exitCode=0 Sep 12 02:03:25.375101 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.375011 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d5764ff8-7b87c99754-8ndt2" event={"ID":"7697cc26-c2ff-46d8-bc9b-2ac5a72328ba","Type":"ContainerDied","Data":"cfb0d6f36636561e5e18e1fd8cf969b61132a8a4daf7653c54357af6cbe3f498"} Sep 12 02:03:25.375101 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.375042 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d5764ff8-7b87c99754-8ndt2" event={"ID":"7697cc26-c2ff-46d8-bc9b-2ac5a72328ba","Type":"ContainerDied","Data":"7dca4f904aa627b7b85656f485c4c48758ba9936ac89d6dc1e240648ce8b04c2"} Sep 12 02:03:25.375101 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.375014 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d5764ff8-7b87c99754-8ndt2" Sep 12 02:03:25.383495 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.383475 2579 scope.go:117] "RemoveContainer" containerID="e2d88c9089531bd0ccd6a798b52227ffc57b176bef5bd72e81e672adcc8e3634" Sep 12 02:03:25.383982 ip-10-0-133-79 kubenswrapper[2579]: E0912 02:03:25.383954 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2d88c9089531bd0ccd6a798b52227ffc57b176bef5bd72e81e672adcc8e3634\": container with ID starting with e2d88c9089531bd0ccd6a798b52227ffc57b176bef5bd72e81e672adcc8e3634 not found: ID does not exist" containerID="e2d88c9089531bd0ccd6a798b52227ffc57b176bef5bd72e81e672adcc8e3634" Sep 12 02:03:25.384071 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.383994 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2d88c9089531bd0ccd6a798b52227ffc57b176bef5bd72e81e672adcc8e3634"} err="failed to get container status \"e2d88c9089531bd0ccd6a798b52227ffc57b176bef5bd72e81e672adcc8e3634\": rpc error: code = NotFound desc = could not find container \"e2d88c9089531bd0ccd6a798b52227ffc57b176bef5bd72e81e672adcc8e3634\": container with ID starting with e2d88c9089531bd0ccd6a798b52227ffc57b176bef5bd72e81e672adcc8e3634 not found: ID does not exist" Sep 12 02:03:25.384071 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.384028 2579 scope.go:117] "RemoveContainer" containerID="686b4150190c3f265df98208b6bcaa4febe94ffdaa6d652aa45b6beb0da57747" Sep 12 02:03:25.395286 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.395266 2579 scope.go:117] "RemoveContainer" containerID="686b4150190c3f265df98208b6bcaa4febe94ffdaa6d652aa45b6beb0da57747" Sep 12 02:03:25.395626 ip-10-0-133-79 kubenswrapper[2579]: E0912 02:03:25.395604 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"686b4150190c3f265df98208b6bcaa4febe94ffdaa6d652aa45b6beb0da57747\": container with ID starting with 686b4150190c3f265df98208b6bcaa4febe94ffdaa6d652aa45b6beb0da57747 not found: ID does not exist" containerID="686b4150190c3f265df98208b6bcaa4febe94ffdaa6d652aa45b6beb0da57747" Sep 12 02:03:25.395722 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.395648 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"686b4150190c3f265df98208b6bcaa4febe94ffdaa6d652aa45b6beb0da57747"} err="failed to get container status \"686b4150190c3f265df98208b6bcaa4febe94ffdaa6d652aa45b6beb0da57747\": rpc error: code = NotFound desc = could not find container \"686b4150190c3f265df98208b6bcaa4febe94ffdaa6d652aa45b6beb0da57747\": container with ID starting with 686b4150190c3f265df98208b6bcaa4febe94ffdaa6d652aa45b6beb0da57747 not found: ID does not exist" Sep 12 02:03:25.395722 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.395672 2579 scope.go:117] "RemoveContainer" containerID="859692dfad20e815545a1db0d9cb6199c432f4876e3ed7406de9273f9fe51280" Sep 12 02:03:25.398945 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.398919 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b838b7-cc4d5586-z6vzt"] Sep 12 02:03:25.402338 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.402319 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b838b7-cc4d5586-z6vzt"] Sep 12 02:03:25.405461 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.405433 2579 scope.go:117] "RemoveContainer" containerID="859692dfad20e815545a1db0d9cb6199c432f4876e3ed7406de9273f9fe51280" Sep 12 02:03:25.405740 ip-10-0-133-79 kubenswrapper[2579]: E0912 02:03:25.405721 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"859692dfad20e815545a1db0d9cb6199c432f4876e3ed7406de9273f9fe51280\": container with ID starting with 859692dfad20e815545a1db0d9cb6199c432f4876e3ed7406de9273f9fe51280 not found: ID does not exist" containerID="859692dfad20e815545a1db0d9cb6199c432f4876e3ed7406de9273f9fe51280" Sep 12 02:03:25.405801 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.405747 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"859692dfad20e815545a1db0d9cb6199c432f4876e3ed7406de9273f9fe51280"} err="failed to get container status \"859692dfad20e815545a1db0d9cb6199c432f4876e3ed7406de9273f9fe51280\": rpc error: code = NotFound desc = could not find container \"859692dfad20e815545a1db0d9cb6199c432f4876e3ed7406de9273f9fe51280\": container with ID starting with 859692dfad20e815545a1db0d9cb6199c432f4876e3ed7406de9273f9fe51280 not found: ID does not exist" Sep 12 02:03:25.405801 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.405763 2579 scope.go:117] "RemoveContainer" containerID="cfb0d6f36636561e5e18e1fd8cf969b61132a8a4daf7653c54357af6cbe3f498" Sep 12 02:03:25.409233 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.409213 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7697cc26-c2ff-46d8-bc9b-2ac5a72328ba-plugins-config-volume\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:03:25.409299 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.409238 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8f7g\" (UniqueName: \"kubernetes.io/projected/7697cc26-c2ff-46d8-bc9b-2ac5a72328ba-kube-api-access-m8f7g\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:03:25.415557 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.415539 2579 scope.go:117] "RemoveContainer" containerID="cfb0d6f36636561e5e18e1fd8cf969b61132a8a4daf7653c54357af6cbe3f498" Sep 12 02:03:25.415685 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.415662 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b838b7-8588bdf7cd-xmndp"] Sep 12 02:03:25.415835 ip-10-0-133-79 kubenswrapper[2579]: E0912 02:03:25.415816 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfb0d6f36636561e5e18e1fd8cf969b61132a8a4daf7653c54357af6cbe3f498\": container with ID starting with cfb0d6f36636561e5e18e1fd8cf969b61132a8a4daf7653c54357af6cbe3f498 not found: ID does not exist" containerID="cfb0d6f36636561e5e18e1fd8cf969b61132a8a4daf7653c54357af6cbe3f498" Sep 12 02:03:25.415894 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.415847 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfb0d6f36636561e5e18e1fd8cf969b61132a8a4daf7653c54357af6cbe3f498"} err="failed to get container status \"cfb0d6f36636561e5e18e1fd8cf969b61132a8a4daf7653c54357af6cbe3f498\": rpc error: code = NotFound desc = could not find container \"cfb0d6f36636561e5e18e1fd8cf969b61132a8a4daf7653c54357af6cbe3f498\": container with ID starting with cfb0d6f36636561e5e18e1fd8cf969b61132a8a4daf7653c54357af6cbe3f498 not found: ID does not exist" Sep 12 02:03:25.419195 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.419174 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b838b7-8588bdf7cd-xmndp"] Sep 12 02:03:25.431337 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.431303 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d5764ff8-7b87c99754-8ndt2"] Sep 12 02:03:25.436006 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.435979 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d5764ff8-7b87c99754-8ndt2"] Sep 12 02:03:25.449135 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.449108 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d5764ff8-747cdc84b5-vlcbm"] Sep 12 02:03:25.451079 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.451057 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d5764ff8-747cdc84b5-vlcbm"] Sep 12 02:03:25.451717 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.451694 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:25.552116 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.552025 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:25.640244 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.640212 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:25.727885 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.727856 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5052265f-0402-428d-bf38-51f62c950021" path="/var/lib/kubelet/pods/5052265f-0402-428d-bf38-51f62c950021/volumes" Sep 12 02:03:25.728328 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.728297 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6eed52bc-6546-4ed3-9c97-fd34057897a7" path="/var/lib/kubelet/pods/6eed52bc-6546-4ed3-9c97-fd34057897a7/volumes" Sep 12 02:03:25.728703 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.728687 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7697cc26-c2ff-46d8-bc9b-2ac5a72328ba" path="/var/lib/kubelet/pods/7697cc26-c2ff-46d8-bc9b-2ac5a72328ba/volumes" Sep 12 02:03:25.729438 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:25.729419 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5971011-f9d7-428a-951c-7e1f8484578a" path="/var/lib/kubelet/pods/d5971011-f9d7-428a-951c-7e1f8484578a/volumes" Sep 12 02:03:26.061791 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:26.061758 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:26.872599 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:26.872564 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:27.039273 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:27.039236 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:03:27.733768 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:27.733735 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:28.474788 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:28.474738 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:29.267000 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:29.266942 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:31.312019 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:31.311979 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:31.412399 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:31.412363 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:31.598522 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:31.598430 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:31.769530 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:31.769492 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:31.869309 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:31.869228 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:31.969347 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:31.969311 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:32.066908 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:32.066873 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:32.140142 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:32.140109 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:32.457928 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:32.457838 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:32.549872 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:32.549841 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:32.719357 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:32.719273 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:32.999172 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:32.999090 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:33.847992 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:33.847959 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:33.946605 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:33.946572 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:34.108324 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:34.108234 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:36.182896 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:36.182848 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:37.239461 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:37.239414 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:38.725148 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:38.725118 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:38.781534 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:38.781497 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:38.910715 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:38.910679 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:39.061191 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:39.061098 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:39.161920 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:39.161884 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:44.807891 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:44.807851 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:44.908441 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:44.908407 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:45.736429 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:45.736390 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:46.099977 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:46.099877 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:46.200055 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:46.200015 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:48.026953 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:48.026919 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:48.763926 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:48.763880 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:48.909229 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:48.909193 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:49.665002 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:49.664965 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:50.462582 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:50.462536 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:51.076652 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:51.076609 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:51.853602 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:51.853574 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e8046f-66dc7c9f5d-vc894"] Sep 12 02:03:51.854052 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:51.854036 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5971011-f9d7-428a-951c-7e1f8484578a" containerName="maas-api" Sep 12 02:03:51.854152 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:51.854056 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5971011-f9d7-428a-951c-7e1f8484578a" containerName="maas-api" Sep 12 02:03:51.854152 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:51.854076 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7697cc26-c2ff-46d8-bc9b-2ac5a72328ba" containerName="payload-pre-processing" Sep 12 02:03:51.854152 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:51.854086 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7697cc26-c2ff-46d8-bc9b-2ac5a72328ba" containerName="payload-pre-processing" Sep 12 02:03:51.854152 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:51.854096 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6eed52bc-6546-4ed3-9c97-fd34057897a7" containerName="payload-processing" Sep 12 02:03:51.854152 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:51.854104 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6eed52bc-6546-4ed3-9c97-fd34057897a7" containerName="payload-processing" Sep 12 02:03:51.854152 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:51.854140 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5052265f-0402-428d-bf38-51f62c950021" containerName="payload-pre-processing" Sep 12 02:03:51.854152 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:51.854148 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5052265f-0402-428d-bf38-51f62c950021" containerName="payload-pre-processing" Sep 12 02:03:51.854504 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:51.854260 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7697cc26-c2ff-46d8-bc9b-2ac5a72328ba" containerName="payload-pre-processing" Sep 12 02:03:51.854504 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:51.854277 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6eed52bc-6546-4ed3-9c97-fd34057897a7" containerName="payload-processing" Sep 12 02:03:51.854504 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:51.854288 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5971011-f9d7-428a-951c-7e1f8484578a" containerName="maas-api" Sep 12 02:03:51.854504 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:51.854298 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5052265f-0402-428d-bf38-51f62c950021" containerName="payload-pre-processing" Sep 12 02:03:51.859545 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:51.859524 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e8046f-66dc7c9f5d-vc894" Sep 12 02:03:51.862661 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:51.862638 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-e8046f-serving-cert\"" Sep 12 02:03:51.867003 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:51.866977 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e8046f-66dc7c9f5d-vc894"] Sep 12 02:03:51.886140 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:51.886104 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:51.888859 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:51.888832 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c105641-487e-4a89-a4d5-a4c2b9206411-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-e8046f-66dc7c9f5d-vc894\" (UID: \"1c105641-487e-4a89-a4d5-a4c2b9206411\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e8046f-66dc7c9f5d-vc894" Sep 12 02:03:51.888975 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:51.888935 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c105641-487e-4a89-a4d5-a4c2b9206411-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-e8046f-66dc7c9f5d-vc894\" (UID: \"1c105641-487e-4a89-a4d5-a4c2b9206411\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e8046f-66dc7c9f5d-vc894" Sep 12 02:03:51.888975 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:51.888959 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c105641-487e-4a89-a4d5-a4c2b9206411-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-e8046f-66dc7c9f5d-vc894\" (UID: \"1c105641-487e-4a89-a4d5-a4c2b9206411\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e8046f-66dc7c9f5d-vc894" Sep 12 02:03:51.889060 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:51.889021 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs4gg\" (UniqueName: \"kubernetes.io/projected/1c105641-487e-4a89-a4d5-a4c2b9206411-kube-api-access-xs4gg\") pod \"maas-api-e2e-shared-a-w2-e8046f-66dc7c9f5d-vc894\" (UID: \"1c105641-487e-4a89-a4d5-a4c2b9206411\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e8046f-66dc7c9f5d-vc894" Sep 12 02:03:51.963716 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:51.963683 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:51.990151 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:51.990117 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs4gg\" (UniqueName: \"kubernetes.io/projected/1c105641-487e-4a89-a4d5-a4c2b9206411-kube-api-access-xs4gg\") pod \"maas-api-e2e-shared-a-w2-e8046f-66dc7c9f5d-vc894\" (UID: \"1c105641-487e-4a89-a4d5-a4c2b9206411\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e8046f-66dc7c9f5d-vc894" Sep 12 02:03:51.990336 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:51.990166 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c105641-487e-4a89-a4d5-a4c2b9206411-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-e8046f-66dc7c9f5d-vc894\" (UID: \"1c105641-487e-4a89-a4d5-a4c2b9206411\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e8046f-66dc7c9f5d-vc894" Sep 12 02:03:51.990336 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:51.990223 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c105641-487e-4a89-a4d5-a4c2b9206411-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-e8046f-66dc7c9f5d-vc894\" (UID: \"1c105641-487e-4a89-a4d5-a4c2b9206411\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e8046f-66dc7c9f5d-vc894" Sep 12 02:03:51.990336 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:51.990240 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c105641-487e-4a89-a4d5-a4c2b9206411-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-e8046f-66dc7c9f5d-vc894\" (UID: \"1c105641-487e-4a89-a4d5-a4c2b9206411\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e8046f-66dc7c9f5d-vc894" Sep 12 02:03:51.990563 ip-10-0-133-79 kubenswrapper[2579]: E0912 02:03:51.990379 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e8046f-serving-cert: secret "maas-api-e2e-shared-a-w2-e8046f-serving-cert" not found Sep 12 02:03:51.990563 ip-10-0-133-79 kubenswrapper[2579]: E0912 02:03:51.990485 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c105641-487e-4a89-a4d5-a4c2b9206411-maas-api-tls podName:1c105641-487e-4a89-a4d5-a4c2b9206411 nodeName:}" failed. No retries permitted until 2026-09-12 02:03:52.490435026 +0000 UTC m=+1135.282428819 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1c105641-487e-4a89-a4d5-a4c2b9206411-maas-api-tls") pod "maas-api-e2e-shared-a-w2-e8046f-66dc7c9f5d-vc894" (UID: "1c105641-487e-4a89-a4d5-a4c2b9206411") : secret "maas-api-e2e-shared-a-w2-e8046f-serving-cert" not found Sep 12 02:03:51.990868 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:51.990849 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c105641-487e-4a89-a4d5-a4c2b9206411-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-e8046f-66dc7c9f5d-vc894\" (UID: \"1c105641-487e-4a89-a4d5-a4c2b9206411\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e8046f-66dc7c9f5d-vc894" Sep 12 02:03:51.992864 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:51.992847 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c105641-487e-4a89-a4d5-a4c2b9206411-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-e8046f-66dc7c9f5d-vc894\" (UID: \"1c105641-487e-4a89-a4d5-a4c2b9206411\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e8046f-66dc7c9f5d-vc894" Sep 12 02:03:52.002633 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:52.002599 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs4gg\" (UniqueName: \"kubernetes.io/projected/1c105641-487e-4a89-a4d5-a4c2b9206411-kube-api-access-xs4gg\") pod \"maas-api-e2e-shared-a-w2-e8046f-66dc7c9f5d-vc894\" (UID: \"1c105641-487e-4a89-a4d5-a4c2b9206411\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e8046f-66dc7c9f5d-vc894" Sep 12 02:03:52.258572 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:52.258542 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:52.477967 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:52.477932 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:52.496188 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:52.496140 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c105641-487e-4a89-a4d5-a4c2b9206411-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-e8046f-66dc7c9f5d-vc894\" (UID: \"1c105641-487e-4a89-a4d5-a4c2b9206411\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e8046f-66dc7c9f5d-vc894" Sep 12 02:03:52.498808 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:52.498779 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c105641-487e-4a89-a4d5-a4c2b9206411-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-e8046f-66dc7c9f5d-vc894\" (UID: \"1c105641-487e-4a89-a4d5-a4c2b9206411\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e8046f-66dc7c9f5d-vc894" Sep 12 02:03:52.577538 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:52.577429 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:52.578850 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:52.578827 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e8046f-55844c5498-pcpbr"] Sep 12 02:03:52.582520 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:52.582502 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e8046f-55844c5498-pcpbr" Sep 12 02:03:52.584809 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:52.584790 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-e8046f-dockercfg-qqkvs\"" Sep 12 02:03:52.585079 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:52.585060 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-e8046f\"" Sep 12 02:03:52.591529 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:52.591501 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e8046f-55844c5498-pcpbr"] Sep 12 02:03:52.698020 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:52.697977 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a007efd1-b179-445a-84ea-28cf8f744c9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-e8046f-55844c5498-pcpbr\" (UID: \"a007efd1-b179-445a-84ea-28cf8f744c9b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e8046f-55844c5498-pcpbr" Sep 12 02:03:52.698189 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:52.698024 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjvr9\" (UniqueName: \"kubernetes.io/projected/a007efd1-b179-445a-84ea-28cf8f744c9b-kube-api-access-jjvr9\") pod \"payload-pre-processing-e2e-shared-a-w2-e8046f-55844c5498-pcpbr\" (UID: \"a007efd1-b179-445a-84ea-28cf8f744c9b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e8046f-55844c5498-pcpbr" Sep 12 02:03:52.771903 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:52.771860 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e8046f-66dc7c9f5d-vc894" Sep 12 02:03:52.799343 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:52.799305 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a007efd1-b179-445a-84ea-28cf8f744c9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-e8046f-55844c5498-pcpbr\" (UID: \"a007efd1-b179-445a-84ea-28cf8f744c9b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e8046f-55844c5498-pcpbr" Sep 12 02:03:52.799582 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:52.799356 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjvr9\" (UniqueName: \"kubernetes.io/projected/a007efd1-b179-445a-84ea-28cf8f744c9b-kube-api-access-jjvr9\") pod \"payload-pre-processing-e2e-shared-a-w2-e8046f-55844c5498-pcpbr\" (UID: \"a007efd1-b179-445a-84ea-28cf8f744c9b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e8046f-55844c5498-pcpbr" Sep 12 02:03:52.800194 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:52.800162 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a007efd1-b179-445a-84ea-28cf8f744c9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-e8046f-55844c5498-pcpbr\" (UID: \"a007efd1-b179-445a-84ea-28cf8f744c9b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e8046f-55844c5498-pcpbr" Sep 12 02:03:52.807531 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:52.807504 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjvr9\" (UniqueName: \"kubernetes.io/projected/a007efd1-b179-445a-84ea-28cf8f744c9b-kube-api-access-jjvr9\") pod \"payload-pre-processing-e2e-shared-a-w2-e8046f-55844c5498-pcpbr\" (UID: \"a007efd1-b179-445a-84ea-28cf8f744c9b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e8046f-55844c5498-pcpbr" Sep 12 02:03:52.896800 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:52.896761 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e8046f-55844c5498-pcpbr" Sep 12 02:03:53.044110 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:53.044086 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e8046f-66dc7c9f5d-vc894"] Sep 12 02:03:53.046256 ip-10-0-133-79 kubenswrapper[2579]: W0912 02:03:53.046222 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c105641_487e_4a89_a4d5_a4c2b9206411.slice/crio-525f2579b5a4608264e14598b8643147ee326bcea1fdc65c1f8b2b0c48ccd030 WatchSource:0}: Error finding container 525f2579b5a4608264e14598b8643147ee326bcea1fdc65c1f8b2b0c48ccd030: Status 404 returned error can't find the container with id 525f2579b5a4608264e14598b8643147ee326bcea1fdc65c1f8b2b0c48ccd030 Sep 12 02:03:53.168084 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:53.168061 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e8046f-55844c5498-pcpbr"] Sep 12 02:03:53.169955 ip-10-0-133-79 kubenswrapper[2579]: W0912 02:03:53.169930 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda007efd1_b179_445a_84ea_28cf8f744c9b.slice/crio-2f7bb4ca668a54cafc85c7af68c7156578033d291c7cfc56772772cbf9142be2 WatchSource:0}: Error finding container 2f7bb4ca668a54cafc85c7af68c7156578033d291c7cfc56772772cbf9142be2: Status 404 returned error can't find the container with id 2f7bb4ca668a54cafc85c7af68c7156578033d291c7cfc56772772cbf9142be2 Sep 12 02:03:53.242863 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:53.242826 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:53.538197 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:53.538109 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e8046f-55844c5498-pcpbr" event={"ID":"a007efd1-b179-445a-84ea-28cf8f744c9b","Type":"ContainerStarted","Data":"d389cd9b85386df3a3bd39a9cf664c6d0ecb4e3e63a9c185d0f794eea9a99cae"} Sep 12 02:03:53.538643 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:53.538284 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e8046f-55844c5498-pcpbr" Sep 12 02:03:53.538643 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:53.538323 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e8046f-55844c5498-pcpbr" event={"ID":"a007efd1-b179-445a-84ea-28cf8f744c9b","Type":"ContainerStarted","Data":"2f7bb4ca668a54cafc85c7af68c7156578033d291c7cfc56772772cbf9142be2"} Sep 12 02:03:53.539442 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:53.539416 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e8046f-66dc7c9f5d-vc894" event={"ID":"1c105641-487e-4a89-a4d5-a4c2b9206411","Type":"ContainerStarted","Data":"525f2579b5a4608264e14598b8643147ee326bcea1fdc65c1f8b2b0c48ccd030"} Sep 12 02:03:53.553665 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:53.553614 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e8046f-55844c5498-pcpbr" podStartSLOduration=1.553585158 podStartE2EDuration="1.553585158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:03:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:03:53.552515116 +0000 UTC m=+1136.344508908" watchObservedRunningTime="2026-09-12 02:03:53.553585158 +0000 UTC m=+1136.345578955" Sep 12 02:03:54.057139 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:54.057092 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:54.546798 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:54.546762 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e8046f-66dc7c9f5d-vc894" event={"ID":"1c105641-487e-4a89-a4d5-a4c2b9206411","Type":"ContainerStarted","Data":"d4fede751a7fc892d506dbfd27571a82d88e83f4e8ff7ed791d00a12367d2b73"} Sep 12 02:03:54.547289 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:54.546990 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e8046f-66dc7c9f5d-vc894" Sep 12 02:03:54.563529 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:54.563478 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e8046f-66dc7c9f5d-vc894" podStartSLOduration=3.06500133 podStartE2EDuration="3.563432354s" totalImagesPullingTime="498.431024ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:03:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:03:54.562776002 +0000 UTC m=+1137.354769797" watchObservedRunningTime="2026-09-12 02:03:54.563432354 +0000 UTC m=+1137.355426146" Sep 12 02:03:58.701824 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:58.701782 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:03:58.765288 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:03:58.765258 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:04:00.337982 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:00.337943 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:04:00.557966 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:00.557937 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e8046f-66dc7c9f5d-vc894" Sep 12 02:04:04.548797 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:04.548761 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e8046f-55844c5498-pcpbr" Sep 12 02:04:11.218442 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:11.218403 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:04:13.846711 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:13.846675 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:04:13.946838 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:13.946801 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:04:14.178584 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:14.178550 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:04:14.499795 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:14.499706 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:04:14.564767 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:14.564733 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:04:15.034163 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:15.034122 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-3fc028/test-model-3fc028-kserve-556bc88545-wt94k"] Sep 12 02:04:15.078327 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:15.078291 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:04:15.200196 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:15.200154 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:04:15.324432 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:15.324338 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:04:16.073235 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:16.073201 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:04:16.733114 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:16.733072 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:04:17.495273 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:17.495234 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:04:17.999503 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:17.999464 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:04:18.087154 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:18.087124 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:04:18.360196 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:18.360100 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:04:19.031215 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:19.031138 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:04:19.930233 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:19.930196 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:04:20.562962 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:20.562925 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:04:21.220054 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:21.220020 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:04:22.032724 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:22.032687 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ac814d-5d9c46dcff-p9x9x"] Sep 12 02:04:22.037965 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:22.037939 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ac814d-5d9c46dcff-p9x9x" Sep 12 02:04:22.040290 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:22.040268 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-ac814d-serving-cert\"" Sep 12 02:04:22.049346 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:22.049312 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ac814d-5d9c46dcff-p9x9x"] Sep 12 02:04:22.066140 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:22.066098 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:04:22.100759 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:22.100723 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/353e00cb-a98d-49eb-b501-ddd6db06d387-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-ac814d-5d9c46dcff-p9x9x\" (UID: \"353e00cb-a98d-49eb-b501-ddd6db06d387\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ac814d-5d9c46dcff-p9x9x" Sep 12 02:04:22.100972 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:22.100806 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l62q\" (UniqueName: \"kubernetes.io/projected/353e00cb-a98d-49eb-b501-ddd6db06d387-kube-api-access-8l62q\") pod \"maas-api-e2e-shared-b-w2-ac814d-5d9c46dcff-p9x9x\" (UID: \"353e00cb-a98d-49eb-b501-ddd6db06d387\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ac814d-5d9c46dcff-p9x9x" Sep 12 02:04:22.100972 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:22.100927 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/353e00cb-a98d-49eb-b501-ddd6db06d387-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-ac814d-5d9c46dcff-p9x9x\" (UID: \"353e00cb-a98d-49eb-b501-ddd6db06d387\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ac814d-5d9c46dcff-p9x9x" Sep 12 02:04:22.101097 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:22.100980 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/353e00cb-a98d-49eb-b501-ddd6db06d387-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-ac814d-5d9c46dcff-p9x9x\" (UID: \"353e00cb-a98d-49eb-b501-ddd6db06d387\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ac814d-5d9c46dcff-p9x9x" Sep 12 02:04:22.127799 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:22.127759 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:04:22.201989 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:22.201949 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/353e00cb-a98d-49eb-b501-ddd6db06d387-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-ac814d-5d9c46dcff-p9x9x\" (UID: \"353e00cb-a98d-49eb-b501-ddd6db06d387\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ac814d-5d9c46dcff-p9x9x" Sep 12 02:04:22.201989 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:22.201991 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/353e00cb-a98d-49eb-b501-ddd6db06d387-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-ac814d-5d9c46dcff-p9x9x\" (UID: \"353e00cb-a98d-49eb-b501-ddd6db06d387\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ac814d-5d9c46dcff-p9x9x" Sep 12 02:04:22.202251 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:22.202019 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/353e00cb-a98d-49eb-b501-ddd6db06d387-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-ac814d-5d9c46dcff-p9x9x\" (UID: \"353e00cb-a98d-49eb-b501-ddd6db06d387\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ac814d-5d9c46dcff-p9x9x" Sep 12 02:04:22.202251 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:22.202077 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l62q\" (UniqueName: \"kubernetes.io/projected/353e00cb-a98d-49eb-b501-ddd6db06d387-kube-api-access-8l62q\") pod \"maas-api-e2e-shared-b-w2-ac814d-5d9c46dcff-p9x9x\" (UID: \"353e00cb-a98d-49eb-b501-ddd6db06d387\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ac814d-5d9c46dcff-p9x9x" Sep 12 02:04:22.202251 ip-10-0-133-79 kubenswrapper[2579]: E0912 02:04:22.202104 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ac814d-serving-cert: secret "maas-api-e2e-shared-b-w2-ac814d-serving-cert" not found Sep 12 02:04:22.202251 ip-10-0-133-79 kubenswrapper[2579]: E0912 02:04:22.202178 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/353e00cb-a98d-49eb-b501-ddd6db06d387-maas-api-tls podName:353e00cb-a98d-49eb-b501-ddd6db06d387 nodeName:}" failed. No retries permitted until 2026-09-12 02:04:22.702160264 +0000 UTC m=+1165.494154053 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/353e00cb-a98d-49eb-b501-ddd6db06d387-maas-api-tls") pod "maas-api-e2e-shared-b-w2-ac814d-5d9c46dcff-p9x9x" (UID: "353e00cb-a98d-49eb-b501-ddd6db06d387") : secret "maas-api-e2e-shared-b-w2-ac814d-serving-cert" not found Sep 12 02:04:22.202743 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:22.202721 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/353e00cb-a98d-49eb-b501-ddd6db06d387-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-ac814d-5d9c46dcff-p9x9x\" (UID: \"353e00cb-a98d-49eb-b501-ddd6db06d387\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ac814d-5d9c46dcff-p9x9x" Sep 12 02:04:22.204709 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:22.204686 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/353e00cb-a98d-49eb-b501-ddd6db06d387-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-ac814d-5d9c46dcff-p9x9x\" (UID: \"353e00cb-a98d-49eb-b501-ddd6db06d387\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ac814d-5d9c46dcff-p9x9x" Sep 12 02:04:22.210899 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:22.210871 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l62q\" (UniqueName: \"kubernetes.io/projected/353e00cb-a98d-49eb-b501-ddd6db06d387-kube-api-access-8l62q\") pod \"maas-api-e2e-shared-b-w2-ac814d-5d9c46dcff-p9x9x\" (UID: \"353e00cb-a98d-49eb-b501-ddd6db06d387\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ac814d-5d9c46dcff-p9x9x" Sep 12 02:04:22.447914 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:22.447875 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:04:22.607976 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:22.607941 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:04:22.706750 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:22.706650 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/353e00cb-a98d-49eb-b501-ddd6db06d387-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-ac814d-5d9c46dcff-p9x9x\" (UID: \"353e00cb-a98d-49eb-b501-ddd6db06d387\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ac814d-5d9c46dcff-p9x9x" Sep 12 02:04:22.707272 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:22.707240 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:04:22.709555 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:22.709531 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/353e00cb-a98d-49eb-b501-ddd6db06d387-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-ac814d-5d9c46dcff-p9x9x\" (UID: \"353e00cb-a98d-49eb-b501-ddd6db06d387\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ac814d-5d9c46dcff-p9x9x" Sep 12 02:04:22.711758 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:22.711737 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ac814d-6c75796f9d-7vsxs"] Sep 12 02:04:22.715483 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:22.715467 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ac814d-6c75796f9d-7vsxs" Sep 12 02:04:22.717981 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:22.717946 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-ac814d\"" Sep 12 02:04:22.717981 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:22.717946 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-ac814d-dockercfg-w46s9\"" Sep 12 02:04:22.728487 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:22.728443 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ac814d-6c75796f9d-7vsxs"] Sep 12 02:04:22.748755 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:22.748716 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ac814d-8d84cc845-8rpfr"] Sep 12 02:04:22.753065 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:22.753045 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ac814d-8d84cc845-8rpfr" Sep 12 02:04:22.762410 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:22.762380 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ac814d-8d84cc845-8rpfr"] Sep 12 02:04:22.807592 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:22.807552 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2513857c-b802-4096-9159-142f7e339886-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-ac814d-6c75796f9d-7vsxs\" (UID: \"2513857c-b802-4096-9159-142f7e339886\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ac814d-6c75796f9d-7vsxs" Sep 12 02:04:22.807592 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:22.807593 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x279s\" (UniqueName: \"kubernetes.io/projected/5f96df74-d12b-488f-8eb6-b8445e716021-kube-api-access-x279s\") pod \"payload-processing-e2e-shared-b-w2-ac814d-8d84cc845-8rpfr\" (UID: \"5f96df74-d12b-488f-8eb6-b8445e716021\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ac814d-8d84cc845-8rpfr" Sep 12 02:04:22.807833 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:22.807699 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rsmb\" (UniqueName: \"kubernetes.io/projected/2513857c-b802-4096-9159-142f7e339886-kube-api-access-2rsmb\") pod \"payload-pre-processing-e2e-shared-b-w2-ac814d-6c75796f9d-7vsxs\" (UID: \"2513857c-b802-4096-9159-142f7e339886\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ac814d-6c75796f9d-7vsxs" Sep 12 02:04:22.807833 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:22.807751 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f96df74-d12b-488f-8eb6-b8445e716021-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-ac814d-8d84cc845-8rpfr\" (UID: \"5f96df74-d12b-488f-8eb6-b8445e716021\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ac814d-8d84cc845-8rpfr" Sep 12 02:04:22.909036 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:22.908996 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2513857c-b802-4096-9159-142f7e339886-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-ac814d-6c75796f9d-7vsxs\" (UID: \"2513857c-b802-4096-9159-142f7e339886\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ac814d-6c75796f9d-7vsxs" Sep 12 02:04:22.909036 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:22.909033 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x279s\" (UniqueName: \"kubernetes.io/projected/5f96df74-d12b-488f-8eb6-b8445e716021-kube-api-access-x279s\") pod \"payload-processing-e2e-shared-b-w2-ac814d-8d84cc845-8rpfr\" (UID: \"5f96df74-d12b-488f-8eb6-b8445e716021\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ac814d-8d84cc845-8rpfr" Sep 12 02:04:22.909320 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:22.909101 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rsmb\" (UniqueName: \"kubernetes.io/projected/2513857c-b802-4096-9159-142f7e339886-kube-api-access-2rsmb\") pod \"payload-pre-processing-e2e-shared-b-w2-ac814d-6c75796f9d-7vsxs\" (UID: \"2513857c-b802-4096-9159-142f7e339886\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ac814d-6c75796f9d-7vsxs" Sep 12 02:04:22.909320 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:22.909138 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f96df74-d12b-488f-8eb6-b8445e716021-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-ac814d-8d84cc845-8rpfr\" (UID: \"5f96df74-d12b-488f-8eb6-b8445e716021\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ac814d-8d84cc845-8rpfr" Sep 12 02:04:22.909848 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:22.909823 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2513857c-b802-4096-9159-142f7e339886-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-ac814d-6c75796f9d-7vsxs\" (UID: \"2513857c-b802-4096-9159-142f7e339886\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ac814d-6c75796f9d-7vsxs" Sep 12 02:04:22.909848 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:22.909842 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f96df74-d12b-488f-8eb6-b8445e716021-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-ac814d-8d84cc845-8rpfr\" (UID: \"5f96df74-d12b-488f-8eb6-b8445e716021\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ac814d-8d84cc845-8rpfr" Sep 12 02:04:22.917659 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:22.917635 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rsmb\" (UniqueName: \"kubernetes.io/projected/2513857c-b802-4096-9159-142f7e339886-kube-api-access-2rsmb\") pod \"payload-pre-processing-e2e-shared-b-w2-ac814d-6c75796f9d-7vsxs\" (UID: \"2513857c-b802-4096-9159-142f7e339886\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ac814d-6c75796f9d-7vsxs" Sep 12 02:04:22.917833 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:22.917813 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x279s\" (UniqueName: \"kubernetes.io/projected/5f96df74-d12b-488f-8eb6-b8445e716021-kube-api-access-x279s\") pod \"payload-processing-e2e-shared-b-w2-ac814d-8d84cc845-8rpfr\" (UID: \"5f96df74-d12b-488f-8eb6-b8445e716021\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ac814d-8d84cc845-8rpfr" Sep 12 02:04:22.950985 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:22.950955 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ac814d-5d9c46dcff-p9x9x" Sep 12 02:04:23.029251 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:23.029221 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ac814d-6c75796f9d-7vsxs" Sep 12 02:04:23.066477 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:23.066421 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ac814d-8d84cc845-8rpfr" Sep 12 02:04:23.223965 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:23.223936 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ac814d-5d9c46dcff-p9x9x"] Sep 12 02:04:23.225843 ip-10-0-133-79 kubenswrapper[2579]: W0912 02:04:23.225815 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod353e00cb_a98d_49eb_b501_ddd6db06d387.slice/crio-31bb878eb3b14a55a7a9932cdced8dce730d8caf837ee36f397b9b52ecb99106 WatchSource:0}: Error finding container 31bb878eb3b14a55a7a9932cdced8dce730d8caf837ee36f397b9b52ecb99106: Status 404 returned error can't find the container with id 31bb878eb3b14a55a7a9932cdced8dce730d8caf837ee36f397b9b52ecb99106 Sep 12 02:04:23.308236 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:23.308211 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ac814d-6c75796f9d-7vsxs"] Sep 12 02:04:23.310060 ip-10-0-133-79 kubenswrapper[2579]: W0912 02:04:23.310020 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2513857c_b802_4096_9159_142f7e339886.slice/crio-42a32aaa8bdb5f775c8fe800be91ef130efe58e2248c1ae79a98455129f2c61e WatchSource:0}: Error finding container 42a32aaa8bdb5f775c8fe800be91ef130efe58e2248c1ae79a98455129f2c61e: Status 404 returned error can't find the container with id 42a32aaa8bdb5f775c8fe800be91ef130efe58e2248c1ae79a98455129f2c61e Sep 12 02:04:23.347529 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:23.347499 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ac814d-8d84cc845-8rpfr"] Sep 12 02:04:23.350766 ip-10-0-133-79 kubenswrapper[2579]: W0912 02:04:23.350734 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f96df74_d12b_488f_8eb6_b8445e716021.slice/crio-dc608dbbdbc3363b7e0336cc0f5506b12b31168f9d339c33786c01480ec069a3 WatchSource:0}: Error finding container dc608dbbdbc3363b7e0336cc0f5506b12b31168f9d339c33786c01480ec069a3: Status 404 returned error can't find the container with id dc608dbbdbc3363b7e0336cc0f5506b12b31168f9d339c33786c01480ec069a3 Sep 12 02:04:23.706551 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:23.706498 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ac814d-8d84cc845-8rpfr" event={"ID":"5f96df74-d12b-488f-8eb6-b8445e716021","Type":"ContainerStarted","Data":"a33528f7cf019f3ae7f2efd6c981f0e0ccaff65308deeb16a3a084f04e61b96e"} Sep 12 02:04:23.706750 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:23.706564 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ac814d-8d84cc845-8rpfr" event={"ID":"5f96df74-d12b-488f-8eb6-b8445e716021","Type":"ContainerStarted","Data":"dc608dbbdbc3363b7e0336cc0f5506b12b31168f9d339c33786c01480ec069a3"} Sep 12 02:04:23.706750 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:23.706597 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ac814d-8d84cc845-8rpfr" Sep 12 02:04:23.708243 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:23.708219 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ac814d-5d9c46dcff-p9x9x" event={"ID":"353e00cb-a98d-49eb-b501-ddd6db06d387","Type":"ContainerStarted","Data":"580374478daea9b43b512cd7a5d61c20e1b15fa008dd57669cf12b0a4e591e8b"} Sep 12 02:04:23.708369 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:23.708262 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ac814d-5d9c46dcff-p9x9x" event={"ID":"353e00cb-a98d-49eb-b501-ddd6db06d387","Type":"ContainerStarted","Data":"31bb878eb3b14a55a7a9932cdced8dce730d8caf837ee36f397b9b52ecb99106"} Sep 12 02:04:23.708369 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:23.708307 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ac814d-5d9c46dcff-p9x9x" Sep 12 02:04:23.710070 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:23.710047 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ac814d-6c75796f9d-7vsxs" event={"ID":"2513857c-b802-4096-9159-142f7e339886","Type":"ContainerStarted","Data":"8bd61707dd1744439d462f89ae5a6b5ff97c1abf6eb046fab547f746b71e2fc4"} Sep 12 02:04:23.710202 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:23.710086 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ac814d-6c75796f9d-7vsxs" event={"ID":"2513857c-b802-4096-9159-142f7e339886","Type":"ContainerStarted","Data":"42a32aaa8bdb5f775c8fe800be91ef130efe58e2248c1ae79a98455129f2c61e"} Sep 12 02:04:23.710202 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:23.710180 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ac814d-6c75796f9d-7vsxs" Sep 12 02:04:23.724082 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:23.724030 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ac814d-8d84cc845-8rpfr" podStartSLOduration=1.724014998 podStartE2EDuration="1.724014998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:04:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:04:23.722331087 +0000 UTC m=+1166.514324883" watchObservedRunningTime="2026-09-12 02:04:23.724014998 +0000 UTC m=+1166.516008809" Sep 12 02:04:23.739071 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:23.739019 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ac814d-6c75796f9d-7vsxs" podStartSLOduration=1.7389933690000001 podStartE2EDuration="1.738993369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:04:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:04:23.736578639 +0000 UTC m=+1166.528572449" watchObservedRunningTime="2026-09-12 02:04:23.738993369 +0000 UTC m=+1166.530987167" Sep 12 02:04:23.754874 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:23.754825 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ac814d-5d9c46dcff-p9x9x" podStartSLOduration=1.440388016 podStartE2EDuration="1.754808453s" totalImagesPullingTime="314.420437ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:04:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:04:23.753539318 +0000 UTC m=+1166.545533116" watchObservedRunningTime="2026-09-12 02:04:23.754808453 +0000 UTC m=+1166.546802242" Sep 12 02:04:28.718533 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:28.718496 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:04:29.727329 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:29.727300 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ac814d-5d9c46dcff-p9x9x" Sep 12 02:04:30.054768 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:30.054670 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-3fc028/test-model-3fc028-kserve-556bc88545-wt94k" podUID="127d7708-4a3f-4ad1-b90f-bf552a9a0afb" containerName="main" containerID="cri-o://5dd7ace081fbd5524a8e92f7c6a3d73d19621d48a703dd4f1ca67450efbb899f" gracePeriod=45 Sep 12 02:04:30.300331 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:30.300306 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3fc028/test-model-3fc028-kserve-556bc88545-wt94k" Sep 12 02:04:30.385800 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:30.385770 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-psmzs\" (UniqueName: \"kubernetes.io/projected/127d7708-4a3f-4ad1-b90f-bf552a9a0afb-kube-api-access-psmzs\") pod \"127d7708-4a3f-4ad1-b90f-bf552a9a0afb\" (UID: \"127d7708-4a3f-4ad1-b90f-bf552a9a0afb\") " Sep 12 02:04:30.385989 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:30.385823 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/127d7708-4a3f-4ad1-b90f-bf552a9a0afb-dshm\") pod \"127d7708-4a3f-4ad1-b90f-bf552a9a0afb\" (UID: \"127d7708-4a3f-4ad1-b90f-bf552a9a0afb\") " Sep 12 02:04:30.385989 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:30.385855 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/127d7708-4a3f-4ad1-b90f-bf552a9a0afb-tmp-dir\") pod \"127d7708-4a3f-4ad1-b90f-bf552a9a0afb\" (UID: \"127d7708-4a3f-4ad1-b90f-bf552a9a0afb\") " Sep 12 02:04:30.385989 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:30.385889 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/127d7708-4a3f-4ad1-b90f-bf552a9a0afb-tls-certs\") pod \"127d7708-4a3f-4ad1-b90f-bf552a9a0afb\" (UID: \"127d7708-4a3f-4ad1-b90f-bf552a9a0afb\") " Sep 12 02:04:30.385989 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:30.385924 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/127d7708-4a3f-4ad1-b90f-bf552a9a0afb-home\") pod \"127d7708-4a3f-4ad1-b90f-bf552a9a0afb\" (UID: \"127d7708-4a3f-4ad1-b90f-bf552a9a0afb\") " Sep 12 02:04:30.385989 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:30.385962 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/127d7708-4a3f-4ad1-b90f-bf552a9a0afb-model-cache\") pod \"127d7708-4a3f-4ad1-b90f-bf552a9a0afb\" (UID: \"127d7708-4a3f-4ad1-b90f-bf552a9a0afb\") " Sep 12 02:04:30.386267 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:30.386163 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/127d7708-4a3f-4ad1-b90f-bf552a9a0afb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "127d7708-4a3f-4ad1-b90f-bf552a9a0afb" (UID: "127d7708-4a3f-4ad1-b90f-bf552a9a0afb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 12 02:04:30.386267 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:30.386173 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/127d7708-4a3f-4ad1-b90f-bf552a9a0afb-home" (OuterVolumeSpecName: "home") pod "127d7708-4a3f-4ad1-b90f-bf552a9a0afb" (UID: "127d7708-4a3f-4ad1-b90f-bf552a9a0afb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 12 02:04:30.386267 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:30.386250 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/127d7708-4a3f-4ad1-b90f-bf552a9a0afb-model-cache" (OuterVolumeSpecName: "model-cache") pod "127d7708-4a3f-4ad1-b90f-bf552a9a0afb" (UID: "127d7708-4a3f-4ad1-b90f-bf552a9a0afb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 12 02:04:30.386444 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:30.386428 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/127d7708-4a3f-4ad1-b90f-bf552a9a0afb-tmp-dir\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:04:30.386521 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:30.386446 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/127d7708-4a3f-4ad1-b90f-bf552a9a0afb-home\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:04:30.386521 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:30.386493 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/127d7708-4a3f-4ad1-b90f-bf552a9a0afb-model-cache\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:04:30.388343 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:30.388323 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/127d7708-4a3f-4ad1-b90f-bf552a9a0afb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "127d7708-4a3f-4ad1-b90f-bf552a9a0afb" (UID: "127d7708-4a3f-4ad1-b90f-bf552a9a0afb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 12 02:04:30.388646 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:30.388624 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/127d7708-4a3f-4ad1-b90f-bf552a9a0afb-dshm" (OuterVolumeSpecName: "dshm") pod "127d7708-4a3f-4ad1-b90f-bf552a9a0afb" (UID: "127d7708-4a3f-4ad1-b90f-bf552a9a0afb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 12 02:04:30.388743 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:30.388643 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/127d7708-4a3f-4ad1-b90f-bf552a9a0afb-kube-api-access-psmzs" (OuterVolumeSpecName: "kube-api-access-psmzs") pod "127d7708-4a3f-4ad1-b90f-bf552a9a0afb" (UID: "127d7708-4a3f-4ad1-b90f-bf552a9a0afb"). InnerVolumeSpecName "kube-api-access-psmzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 12 02:04:30.487531 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:30.487495 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psmzs\" (UniqueName: \"kubernetes.io/projected/127d7708-4a3f-4ad1-b90f-bf552a9a0afb-kube-api-access-psmzs\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:04:30.487531 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:30.487524 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/127d7708-4a3f-4ad1-b90f-bf552a9a0afb-dshm\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:04:30.487531 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:30.487532 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/127d7708-4a3f-4ad1-b90f-bf552a9a0afb-tls-certs\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:04:30.749646 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:30.749552 2579 generic.go:356] "Generic (PLEG): container finished" podID="127d7708-4a3f-4ad1-b90f-bf552a9a0afb" containerID="5dd7ace081fbd5524a8e92f7c6a3d73d19621d48a703dd4f1ca67450efbb899f" exitCode=0 Sep 12 02:04:30.749646 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:30.749592 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3fc028/test-model-3fc028-kserve-556bc88545-wt94k" Sep 12 02:04:30.749646 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:30.749615 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3fc028/test-model-3fc028-kserve-556bc88545-wt94k" event={"ID":"127d7708-4a3f-4ad1-b90f-bf552a9a0afb","Type":"ContainerDied","Data":"5dd7ace081fbd5524a8e92f7c6a3d73d19621d48a703dd4f1ca67450efbb899f"} Sep 12 02:04:30.750122 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:30.749650 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3fc028/test-model-3fc028-kserve-556bc88545-wt94k" event={"ID":"127d7708-4a3f-4ad1-b90f-bf552a9a0afb","Type":"ContainerDied","Data":"6452dddad8f6af369afc21692d58a089b161ef64bd89c36575bd9c1f85eab411"} Sep 12 02:04:30.750122 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:30.749672 2579 scope.go:117] "RemoveContainer" containerID="5dd7ace081fbd5524a8e92f7c6a3d73d19621d48a703dd4f1ca67450efbb899f" Sep 12 02:04:30.759804 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:30.759783 2579 scope.go:117] "RemoveContainer" containerID="5dd7ace081fbd5524a8e92f7c6a3d73d19621d48a703dd4f1ca67450efbb899f" Sep 12 02:04:30.760138 ip-10-0-133-79 kubenswrapper[2579]: E0912 02:04:30.760109 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dd7ace081fbd5524a8e92f7c6a3d73d19621d48a703dd4f1ca67450efbb899f\": container with ID starting with 5dd7ace081fbd5524a8e92f7c6a3d73d19621d48a703dd4f1ca67450efbb899f not found: ID does not exist" containerID="5dd7ace081fbd5524a8e92f7c6a3d73d19621d48a703dd4f1ca67450efbb899f" Sep 12 02:04:30.760214 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:30.760145 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dd7ace081fbd5524a8e92f7c6a3d73d19621d48a703dd4f1ca67450efbb899f"} err="failed to get container status \"5dd7ace081fbd5524a8e92f7c6a3d73d19621d48a703dd4f1ca67450efbb899f\": rpc error: code = NotFound desc = could not find container \"5dd7ace081fbd5524a8e92f7c6a3d73d19621d48a703dd4f1ca67450efbb899f\": container with ID starting with 5dd7ace081fbd5524a8e92f7c6a3d73d19621d48a703dd4f1ca67450efbb899f not found: ID does not exist" Sep 12 02:04:30.772218 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:30.772179 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-3fc028/test-model-3fc028-kserve-556bc88545-wt94k"] Sep 12 02:04:30.774554 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:30.774527 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-3fc028/test-model-3fc028-kserve-556bc88545-wt94k"] Sep 12 02:04:31.727992 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:31.727948 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="127d7708-4a3f-4ad1-b90f-bf552a9a0afb" path="/var/lib/kubelet/pods/127d7708-4a3f-4ad1-b90f-bf552a9a0afb/volumes" Sep 12 02:04:34.718308 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:34.718274 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ac814d-8d84cc845-8rpfr" Sep 12 02:04:34.718726 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:34.718483 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ac814d-6c75796f9d-7vsxs" Sep 12 02:04:40.438714 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:40.438679 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:04:40.633526 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:40.633487 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:04:41.223359 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:41.223322 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:04:42.060204 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:42.060148 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:04:42.819717 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:42.819665 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:04:43.661021 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:43.660979 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:04:44.286830 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:44.286790 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:04:44.929085 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:44.928991 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:04:45.760472 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:45.760418 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:04:46.549610 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:46.549574 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:04:46.778775 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:46.778743 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:04:47.013977 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:47.013937 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:04:47.149070 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:47.149035 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:04:47.251129 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:47.251091 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:04:47.891300 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:47.891256 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:04:48.197529 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:48.197419 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:04:48.749688 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:48.749650 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:04:49.017422 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:49.017342 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8063-50947c"] Sep 12 02:04:49.017863 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:49.017849 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="127d7708-4a3f-4ad1-b90f-bf552a9a0afb" containerName="main" Sep 12 02:04:49.017906 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:49.017865 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="127d7708-4a3f-4ad1-b90f-bf552a9a0afb" containerName="main" Sep 12 02:04:49.017959 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:49.017951 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="127d7708-4a3f-4ad1-b90f-bf552a9a0afb" containerName="main" Sep 12 02:04:49.022313 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:49.022296 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8063-50947c" Sep 12 02:04:49.025078 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:49.025046 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-dng96\"" Sep 12 02:04:49.027614 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:49.027591 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8063-50947c"] Sep 12 02:04:49.188178 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:49.188129 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d99rw\" (UniqueName: \"kubernetes.io/projected/9981ab6e-7a74-49aa-bcf4-36c3bb72aca7-kube-api-access-d99rw\") pod \"test-curl-8063-50947c\" (UID: \"9981ab6e-7a74-49aa-bcf4-36c3bb72aca7\") " pod="opendatahub/test-curl-8063-50947c" Sep 12 02:04:49.289481 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:49.289365 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d99rw\" (UniqueName: \"kubernetes.io/projected/9981ab6e-7a74-49aa-bcf4-36c3bb72aca7-kube-api-access-d99rw\") pod \"test-curl-8063-50947c\" (UID: \"9981ab6e-7a74-49aa-bcf4-36c3bb72aca7\") " pod="opendatahub/test-curl-8063-50947c" Sep 12 02:04:49.298272 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:49.298247 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d99rw\" (UniqueName: \"kubernetes.io/projected/9981ab6e-7a74-49aa-bcf4-36c3bb72aca7-kube-api-access-d99rw\") pod \"test-curl-8063-50947c\" (UID: \"9981ab6e-7a74-49aa-bcf4-36c3bb72aca7\") " pod="opendatahub/test-curl-8063-50947c" Sep 12 02:04:49.333318 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:49.333283 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8063-50947c" Sep 12 02:04:49.402348 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:49.402292 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:04:49.605640 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:49.605612 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8063-50947c"] Sep 12 02:04:49.607242 ip-10-0-133-79 kubenswrapper[2579]: W0912 02:04:49.607214 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9981ab6e_7a74_49aa_bcf4_36c3bb72aca7.slice/crio-8ec91cf50d58e830720181e0394b753a0bbefb82bd703c4195411c3f90ee8561 WatchSource:0}: Error finding container 8ec91cf50d58e830720181e0394b753a0bbefb82bd703c4195411c3f90ee8561: Status 404 returned error can't find the container with id 8ec91cf50d58e830720181e0394b753a0bbefb82bd703c4195411c3f90ee8561 Sep 12 02:04:49.860183 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:49.860086 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8063-50947c" event={"ID":"9981ab6e-7a74-49aa-bcf4-36c3bb72aca7","Type":"ContainerStarted","Data":"8ec91cf50d58e830720181e0394b753a0bbefb82bd703c4195411c3f90ee8561"} Sep 12 02:04:50.507880 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:50.507833 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:04:50.586976 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:50.586942 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3fc028-668b6db875-svkb5"] Sep 12 02:04:50.587250 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:50.587225 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3fc028-668b6db875-svkb5" podUID="268dd63e-c941-4687-ba08-f4e342b2585f" containerName="maas-api" containerID="cri-o://421b529c01a7bd5fb73cddacd7090ea69ee09be71fa25b219e5e31c7aa92712c" gracePeriod=30 Sep 12 02:04:50.617434 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:50.617391 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:04:50.644218 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:50.642113 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3fc028-67db4f49b5-5jjjf"] Sep 12 02:04:50.644218 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:50.642354 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-3fc028-67db4f49b5-5jjjf" podUID="6651f43f-9684-460e-b2f6-787e35ebe96d" containerName="payload-processing" containerID="cri-o://b64484c70e9d4e8cf20ef89b038f12687d0653b7d0cc78ac74d05a5ecb2f92eb" gracePeriod=30 Sep 12 02:04:50.720119 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:50.720043 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:04:50.804663 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:50.804496 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:04:50.862152 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:50.862089 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:04:50.874796 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:50.874761 2579 generic.go:356] "Generic (PLEG): container finished" podID="6651f43f-9684-460e-b2f6-787e35ebe96d" containerID="b64484c70e9d4e8cf20ef89b038f12687d0653b7d0cc78ac74d05a5ecb2f92eb" exitCode=0 Sep 12 02:04:50.874988 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:50.874915 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3fc028-67db4f49b5-5jjjf" event={"ID":"6651f43f-9684-460e-b2f6-787e35ebe96d","Type":"ContainerDied","Data":"b64484c70e9d4e8cf20ef89b038f12687d0653b7d0cc78ac74d05a5ecb2f92eb"} Sep 12 02:04:50.877570 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:50.877492 2579 generic.go:356] "Generic (PLEG): container finished" podID="268dd63e-c941-4687-ba08-f4e342b2585f" containerID="421b529c01a7bd5fb73cddacd7090ea69ee09be71fa25b219e5e31c7aa92712c" exitCode=0 Sep 12 02:04:50.877570 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:50.877567 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3fc028-668b6db875-svkb5" event={"ID":"268dd63e-c941-4687-ba08-f4e342b2585f","Type":"ContainerDied","Data":"421b529c01a7bd5fb73cddacd7090ea69ee09be71fa25b219e5e31c7aa92712c"} Sep 12 02:04:50.943656 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:50.943630 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3fc028-668b6db875-svkb5" Sep 12 02:04:50.946844 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:50.946811 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3fc028-67db4f49b5-5jjjf" Sep 12 02:04:51.109187 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:51.109072 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-894qj\" (UniqueName: \"kubernetes.io/projected/6651f43f-9684-460e-b2f6-787e35ebe96d-kube-api-access-894qj\") pod \"6651f43f-9684-460e-b2f6-787e35ebe96d\" (UID: \"6651f43f-9684-460e-b2f6-787e35ebe96d\") " Sep 12 02:04:51.109187 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:51.109139 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/268dd63e-c941-4687-ba08-f4e342b2585f-metrics-certs\") pod \"268dd63e-c941-4687-ba08-f4e342b2585f\" (UID: \"268dd63e-c941-4687-ba08-f4e342b2585f\") " Sep 12 02:04:51.109581 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:51.109201 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/268dd63e-c941-4687-ba08-f4e342b2585f-maas-api-tls\") pod \"268dd63e-c941-4687-ba08-f4e342b2585f\" (UID: \"268dd63e-c941-4687-ba08-f4e342b2585f\") " Sep 12 02:04:51.109581 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:51.109230 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-txprt\" (UniqueName: \"kubernetes.io/projected/268dd63e-c941-4687-ba08-f4e342b2585f-kube-api-access-txprt\") pod \"268dd63e-c941-4687-ba08-f4e342b2585f\" (UID: \"268dd63e-c941-4687-ba08-f4e342b2585f\") " Sep 12 02:04:51.109581 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:51.109288 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/268dd63e-c941-4687-ba08-f4e342b2585f-openshift-service-ca\") pod \"268dd63e-c941-4687-ba08-f4e342b2585f\" (UID: \"268dd63e-c941-4687-ba08-f4e342b2585f\") " Sep 12 02:04:51.109581 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:51.109353 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6651f43f-9684-460e-b2f6-787e35ebe96d-plugins-config-volume\") pod \"6651f43f-9684-460e-b2f6-787e35ebe96d\" (UID: \"6651f43f-9684-460e-b2f6-787e35ebe96d\") " Sep 12 02:04:51.109840 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:51.109743 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/268dd63e-c941-4687-ba08-f4e342b2585f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "268dd63e-c941-4687-ba08-f4e342b2585f" (UID: "268dd63e-c941-4687-ba08-f4e342b2585f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 12 02:04:51.109895 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:51.109857 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/268dd63e-c941-4687-ba08-f4e342b2585f-openshift-service-ca\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:04:51.110029 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:51.109947 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6651f43f-9684-460e-b2f6-787e35ebe96d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6651f43f-9684-460e-b2f6-787e35ebe96d" (UID: "6651f43f-9684-460e-b2f6-787e35ebe96d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 12 02:04:51.111835 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:51.111805 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6651f43f-9684-460e-b2f6-787e35ebe96d-kube-api-access-894qj" (OuterVolumeSpecName: "kube-api-access-894qj") pod "6651f43f-9684-460e-b2f6-787e35ebe96d" (UID: "6651f43f-9684-460e-b2f6-787e35ebe96d"). InnerVolumeSpecName "kube-api-access-894qj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 12 02:04:51.112254 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:51.112208 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/268dd63e-c941-4687-ba08-f4e342b2585f-kube-api-access-txprt" (OuterVolumeSpecName: "kube-api-access-txprt") pod "268dd63e-c941-4687-ba08-f4e342b2585f" (UID: "268dd63e-c941-4687-ba08-f4e342b2585f"). InnerVolumeSpecName "kube-api-access-txprt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 12 02:04:51.112361 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:51.112302 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/268dd63e-c941-4687-ba08-f4e342b2585f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "268dd63e-c941-4687-ba08-f4e342b2585f" (UID: "268dd63e-c941-4687-ba08-f4e342b2585f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 12 02:04:51.112361 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:51.112312 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/268dd63e-c941-4687-ba08-f4e342b2585f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "268dd63e-c941-4687-ba08-f4e342b2585f" (UID: "268dd63e-c941-4687-ba08-f4e342b2585f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 12 02:04:51.177121 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:51.177078 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:04:51.211207 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:51.211172 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6651f43f-9684-460e-b2f6-787e35ebe96d-plugins-config-volume\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:04:51.211207 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:51.211205 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-894qj\" (UniqueName: \"kubernetes.io/projected/6651f43f-9684-460e-b2f6-787e35ebe96d-kube-api-access-894qj\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:04:51.211408 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:51.211217 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/268dd63e-c941-4687-ba08-f4e342b2585f-metrics-certs\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:04:51.211408 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:51.211233 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/268dd63e-c941-4687-ba08-f4e342b2585f-maas-api-tls\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:04:51.211408 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:51.211243 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txprt\" (UniqueName: \"kubernetes.io/projected/268dd63e-c941-4687-ba08-f4e342b2585f-kube-api-access-txprt\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:04:51.279387 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:51.279354 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:04:51.889431 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:51.889395 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3fc028-668b6db875-svkb5" Sep 12 02:04:51.889838 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:51.889425 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3fc028-668b6db875-svkb5" event={"ID":"268dd63e-c941-4687-ba08-f4e342b2585f","Type":"ContainerDied","Data":"5b38489aff8e57d17ad09d4569f2cfae575073c0dda7a233ab67333fa31f76dd"} Sep 12 02:04:51.889838 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:51.889517 2579 scope.go:117] "RemoveContainer" containerID="421b529c01a7bd5fb73cddacd7090ea69ee09be71fa25b219e5e31c7aa92712c" Sep 12 02:04:51.891263 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:51.891231 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3fc028-67db4f49b5-5jjjf" event={"ID":"6651f43f-9684-460e-b2f6-787e35ebe96d","Type":"ContainerDied","Data":"c049962f9637a48faf4fbe6f6d69d1521cb3d5a7a6924e7ac6be3cec909e72ad"} Sep 12 02:04:51.891369 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:51.891328 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3fc028-67db4f49b5-5jjjf" Sep 12 02:04:51.923873 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:51.923829 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3fc028-67db4f49b5-5jjjf"] Sep 12 02:04:51.937078 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:51.937048 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3fc028-67db4f49b5-5jjjf"] Sep 12 02:04:51.965005 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:51.964975 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3fc028-668b6db875-svkb5"] Sep 12 02:04:51.965633 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:51.965617 2579 scope.go:117] "RemoveContainer" containerID="b64484c70e9d4e8cf20ef89b038f12687d0653b7d0cc78ac74d05a5ecb2f92eb" Sep 12 02:04:51.974017 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:51.973982 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3fc028-668b6db875-svkb5"] Sep 12 02:04:52.106225 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:52.106193 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:04:52.303825 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:52.303734 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:04:52.726087 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:52.726057 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:04:52.908276 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:52.908242 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8063-50947c" event={"ID":"9981ab6e-7a74-49aa-bcf4-36c3bb72aca7","Type":"ContainerStarted","Data":"c1780f39c02963fa9995444db819318ee593c09d6c48a1b87258038f9665d0d3"} Sep 12 02:04:52.930021 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:52.929961 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8063-50947c" podStartSLOduration=1.514436365 podStartE2EDuration="3.929944594s" totalImagesPullingTime="2.415508229s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:04:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:04:52.929392763 +0000 UTC m=+1195.721386557" watchObservedRunningTime="2026-09-12 02:04:52.929944594 +0000 UTC m=+1195.721938388" Sep 12 02:04:53.382139 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:53.382103 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:04:53.545573 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:53.545537 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8063-50947c"] Sep 12 02:04:53.549346 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:53.549311 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8063-50947c"] Sep 12 02:04:53.675517 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:53.675426 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8063-7a2a8d"] Sep 12 02:04:53.675949 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:53.675935 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9981ab6e-7a74-49aa-bcf4-36c3bb72aca7" containerName="test-curl-8063-50947c" Sep 12 02:04:53.675989 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:53.675951 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9981ab6e-7a74-49aa-bcf4-36c3bb72aca7" containerName="test-curl-8063-50947c" Sep 12 02:04:53.675989 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:53.675968 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="268dd63e-c941-4687-ba08-f4e342b2585f" containerName="maas-api" Sep 12 02:04:53.675989 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:53.675974 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="268dd63e-c941-4687-ba08-f4e342b2585f" containerName="maas-api" Sep 12 02:04:53.675989 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:53.675986 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6651f43f-9684-460e-b2f6-787e35ebe96d" containerName="payload-processing" Sep 12 02:04:53.676119 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:53.675991 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6651f43f-9684-460e-b2f6-787e35ebe96d" containerName="payload-processing" Sep 12 02:04:53.676119 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:53.676058 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="268dd63e-c941-4687-ba08-f4e342b2585f" containerName="maas-api" Sep 12 02:04:53.676119 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:53.676067 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6651f43f-9684-460e-b2f6-787e35ebe96d" containerName="payload-processing" Sep 12 02:04:53.676119 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:53.676075 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9981ab6e-7a74-49aa-bcf4-36c3bb72aca7" containerName="test-curl-8063-50947c" Sep 12 02:04:53.679537 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:53.679518 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8063-7a2a8d" Sep 12 02:04:53.683755 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:53.683730 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8063-7a2a8d"] Sep 12 02:04:53.726873 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:53.726828 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="268dd63e-c941-4687-ba08-f4e342b2585f" path="/var/lib/kubelet/pods/268dd63e-c941-4687-ba08-f4e342b2585f/volumes" Sep 12 02:04:53.727299 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:53.727281 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6651f43f-9684-460e-b2f6-787e35ebe96d" path="/var/lib/kubelet/pods/6651f43f-9684-460e-b2f6-787e35ebe96d/volumes" Sep 12 02:04:53.842407 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:53.842361 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nngts\" (UniqueName: \"kubernetes.io/projected/aef019f4-965e-4995-8f32-c53741b3157c-kube-api-access-nngts\") pod \"test-curl-8063-7a2a8d\" (UID: \"aef019f4-965e-4995-8f32-c53741b3157c\") " pod="opendatahub/test-curl-8063-7a2a8d" Sep 12 02:04:53.913427 ip-10-0-133-79 kubenswrapper[2579]: E0912 02:04:53.913391 2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8063-50947c" container="test-curl-8063-50947c" Sep 12 02:04:53.943323 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:53.943238 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nngts\" (UniqueName: \"kubernetes.io/projected/aef019f4-965e-4995-8f32-c53741b3157c-kube-api-access-nngts\") pod \"test-curl-8063-7a2a8d\" (UID: \"aef019f4-965e-4995-8f32-c53741b3157c\") " pod="opendatahub/test-curl-8063-7a2a8d" Sep 12 02:04:53.951198 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:53.951170 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nngts\" (UniqueName: \"kubernetes.io/projected/aef019f4-965e-4995-8f32-c53741b3157c-kube-api-access-nngts\") pod \"test-curl-8063-7a2a8d\" (UID: \"aef019f4-965e-4995-8f32-c53741b3157c\") " pod="opendatahub/test-curl-8063-7a2a8d" Sep 12 02:04:53.990243 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:53.990208 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8063-7a2a8d" Sep 12 02:04:54.148889 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:54.148855 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:04:54.265085 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:54.265059 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8063-7a2a8d"] Sep 12 02:04:54.267201 ip-10-0-133-79 kubenswrapper[2579]: W0912 02:04:54.267173 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaef019f4_965e_4995_8f32_c53741b3157c.slice/crio-fa9784de5df5e09ff6d01577cbfacc4371b948470f644551a298596cff13fce6 WatchSource:0}: Error finding container fa9784de5df5e09ff6d01577cbfacc4371b948470f644551a298596cff13fce6: Status 404 returned error can't find the container with id fa9784de5df5e09ff6d01577cbfacc4371b948470f644551a298596cff13fce6 Sep 12 02:04:54.779001 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:54.778959 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:04:54.920609 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:54.920572 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8063-7a2a8d" event={"ID":"aef019f4-965e-4995-8f32-c53741b3157c","Type":"ContainerStarted","Data":"67dbfa77b78ab30af926af5b5ed017959f9909fe08dbaaaa3608b673876ddf4e"} Sep 12 02:04:54.921050 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:54.920619 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8063-7a2a8d" event={"ID":"aef019f4-965e-4995-8f32-c53741b3157c","Type":"ContainerStarted","Data":"fa9784de5df5e09ff6d01577cbfacc4371b948470f644551a298596cff13fce6"} Sep 12 02:04:54.921050 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:54.920742 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8063-50947c" podUID="9981ab6e-7a74-49aa-bcf4-36c3bb72aca7" containerName="test-curl-8063-50947c" containerID="cri-o://c1780f39c02963fa9995444db819318ee593c09d6c48a1b87258038f9665d0d3" gracePeriod=2 Sep 12 02:04:54.934039 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:54.933986 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8063-7a2a8d" podStartSLOduration=1.933970357 podStartE2EDuration="1.933970357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:04:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:04:54.932839102 +0000 UTC m=+1197.724832898" watchObservedRunningTime="2026-09-12 02:04:54.933970357 +0000 UTC m=+1197.725964195" Sep 12 02:04:55.581730 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:55.581693 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8063-7a2a8d"] Sep 12 02:04:55.586058 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:55.586023 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8063-7a2a8d"] Sep 12 02:04:55.704607 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:55.704565 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8063-be83c6"] Sep 12 02:04:55.705104 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:55.705085 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aef019f4-965e-4995-8f32-c53741b3157c" containerName="test-curl-8063-7a2a8d" Sep 12 02:04:55.705188 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:55.705107 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="aef019f4-965e-4995-8f32-c53741b3157c" containerName="test-curl-8063-7a2a8d" Sep 12 02:04:55.705248 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:55.705230 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="aef019f4-965e-4995-8f32-c53741b3157c" containerName="test-curl-8063-7a2a8d" Sep 12 02:04:55.708371 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:55.708350 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8063-be83c6" Sep 12 02:04:55.713625 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:55.713597 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8063-be83c6"] Sep 12 02:04:55.763163 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:55.763121 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9rqx\" (UniqueName: \"kubernetes.io/projected/33168515-4d1d-4972-a813-1e475ef15769-kube-api-access-s9rqx\") pod \"test-curl-8063-be83c6\" (UID: \"33168515-4d1d-4972-a813-1e475ef15769\") " pod="opendatahub/test-curl-8063-be83c6" Sep 12 02:04:55.864835 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:55.864740 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9rqx\" (UniqueName: \"kubernetes.io/projected/33168515-4d1d-4972-a813-1e475ef15769-kube-api-access-s9rqx\") pod \"test-curl-8063-be83c6\" (UID: \"33168515-4d1d-4972-a813-1e475ef15769\") " pod="opendatahub/test-curl-8063-be83c6" Sep 12 02:04:55.872892 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:55.872866 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9rqx\" (UniqueName: \"kubernetes.io/projected/33168515-4d1d-4972-a813-1e475ef15769-kube-api-access-s9rqx\") pod \"test-curl-8063-be83c6\" (UID: \"33168515-4d1d-4972-a813-1e475ef15769\") " pod="opendatahub/test-curl-8063-be83c6" Sep 12 02:04:55.926533 ip-10-0-133-79 kubenswrapper[2579]: E0912 02:04:55.926506 2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8063-7a2a8d" container="test-curl-8063-7a2a8d" Sep 12 02:04:56.019377 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:56.019336 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8063-be83c6" Sep 12 02:04:56.297321 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:56.297295 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8063-be83c6"] Sep 12 02:04:56.299522 ip-10-0-133-79 kubenswrapper[2579]: W0912 02:04:56.299487 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod33168515_4d1d_4972_a813_1e475ef15769.slice/crio-12ef46358bbcb35dc63c06ad993a23a1c787269fb3cfd0b363a5f2e9d0d6c6c8 WatchSource:0}: Error finding container 12ef46358bbcb35dc63c06ad993a23a1c787269fb3cfd0b363a5f2e9d0d6c6c8: Status 404 returned error can't find the container with id 12ef46358bbcb35dc63c06ad993a23a1c787269fb3cfd0b363a5f2e9d0d6c6c8 Sep 12 02:04:56.934475 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:56.934422 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8063-be83c6" event={"ID":"33168515-4d1d-4972-a813-1e475ef15769","Type":"ContainerStarted","Data":"910ab83d6a828b94415afff6684b80bfe0ef5f11c238ac58ae194b86a4858400"} Sep 12 02:04:56.934857 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:56.934487 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8063-be83c6" event={"ID":"33168515-4d1d-4972-a813-1e475ef15769","Type":"ContainerStarted","Data":"12ef46358bbcb35dc63c06ad993a23a1c787269fb3cfd0b363a5f2e9d0d6c6c8"} Sep 12 02:04:56.934857 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:56.934584 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8063-7a2a8d" podUID="aef019f4-965e-4995-8f32-c53741b3157c" containerName="test-curl-8063-7a2a8d" containerID="cri-o://67dbfa77b78ab30af926af5b5ed017959f9909fe08dbaaaa3608b673876ddf4e" gracePeriod=2 Sep 12 02:04:56.949412 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:56.949355 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8063-be83c6" podStartSLOduration=1.949336263 podStartE2EDuration="1.949336263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:04:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:04:56.947927267 +0000 UTC m=+1199.739921063" watchObservedRunningTime="2026-09-12 02:04:56.949336263 +0000 UTC m=+1199.741330092" Sep 12 02:04:56.970146 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:56.970110 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:04:57.038633 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:57.038604 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:04:57.107233 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:57.107212 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8063-50947c" Sep 12 02:04:57.178052 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:57.177959 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d99rw\" (UniqueName: \"kubernetes.io/projected/9981ab6e-7a74-49aa-bcf4-36c3bb72aca7-kube-api-access-d99rw\") pod \"9981ab6e-7a74-49aa-bcf4-36c3bb72aca7\" (UID: \"9981ab6e-7a74-49aa-bcf4-36c3bb72aca7\") " Sep 12 02:04:57.180938 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:57.180867 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9981ab6e-7a74-49aa-bcf4-36c3bb72aca7-kube-api-access-d99rw" (OuterVolumeSpecName: "kube-api-access-d99rw") pod "9981ab6e-7a74-49aa-bcf4-36c3bb72aca7" (UID: "9981ab6e-7a74-49aa-bcf4-36c3bb72aca7"). InnerVolumeSpecName "kube-api-access-d99rw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 12 02:04:57.247331 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:57.247302 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:04:57.279426 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:57.279386 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d99rw\" (UniqueName: \"kubernetes.io/projected/9981ab6e-7a74-49aa-bcf4-36c3bb72aca7-kube-api-access-d99rw\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:04:57.552949 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:57.552866 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8063-be83c6"] Sep 12 02:04:57.558961 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:57.558932 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8063-be83c6"] Sep 12 02:04:57.689959 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:57.689922 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8063-8be879"] Sep 12 02:04:57.690408 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:57.690395 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33168515-4d1d-4972-a813-1e475ef15769" containerName="test-curl-8063-be83c6" Sep 12 02:04:57.690476 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:57.690410 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="33168515-4d1d-4972-a813-1e475ef15769" containerName="test-curl-8063-be83c6" Sep 12 02:04:57.690517 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:57.690507 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="33168515-4d1d-4972-a813-1e475ef15769" containerName="test-curl-8063-be83c6" Sep 12 02:04:57.693777 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:57.693760 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8063-8be879" Sep 12 02:04:57.698903 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:57.698872 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8063-8be879"] Sep 12 02:04:57.729280 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:57.729239 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9981ab6e-7a74-49aa-bcf4-36c3bb72aca7" path="/var/lib/kubelet/pods/9981ab6e-7a74-49aa-bcf4-36c3bb72aca7/volumes" Sep 12 02:04:57.783415 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:57.783378 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4t69\" (UniqueName: \"kubernetes.io/projected/0f6afdca-e7c6-4bfc-9ffe-9acd05d11e0c-kube-api-access-w4t69\") pod \"test-curl-8063-8be879\" (UID: \"0f6afdca-e7c6-4bfc-9ffe-9acd05d11e0c\") " pod="opendatahub/test-curl-8063-8be879" Sep 12 02:04:57.840865 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:57.840840 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:04:57.884874 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:57.884827 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4t69\" (UniqueName: \"kubernetes.io/projected/0f6afdca-e7c6-4bfc-9ffe-9acd05d11e0c-kube-api-access-w4t69\") pod \"test-curl-8063-8be879\" (UID: \"0f6afdca-e7c6-4bfc-9ffe-9acd05d11e0c\") " pod="opendatahub/test-curl-8063-8be879" Sep 12 02:04:57.893050 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:57.893018 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4t69\" (UniqueName: \"kubernetes.io/projected/0f6afdca-e7c6-4bfc-9ffe-9acd05d11e0c-kube-api-access-w4t69\") pod \"test-curl-8063-8be879\" (UID: \"0f6afdca-e7c6-4bfc-9ffe-9acd05d11e0c\") " pod="opendatahub/test-curl-8063-8be879" Sep 12 02:04:57.940918 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:57.940880 2579 generic.go:356] "Generic (PLEG): container finished" podID="9981ab6e-7a74-49aa-bcf4-36c3bb72aca7" containerID="c1780f39c02963fa9995444db819318ee593c09d6c48a1b87258038f9665d0d3" exitCode=137 Sep 12 02:04:57.941379 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:57.940930 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8063-50947c" Sep 12 02:04:57.941379 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:57.940975 2579 scope.go:117] "RemoveContainer" containerID="c1780f39c02963fa9995444db819318ee593c09d6c48a1b87258038f9665d0d3" Sep 12 02:04:57.941379 ip-10-0-133-79 kubenswrapper[2579]: E0912 02:04:57.941153 2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8063-be83c6" container="test-curl-8063-be83c6" Sep 12 02:04:57.950124 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:57.950102 2579 scope.go:117] "RemoveContainer" containerID="c1780f39c02963fa9995444db819318ee593c09d6c48a1b87258038f9665d0d3" Sep 12 02:04:57.950467 ip-10-0-133-79 kubenswrapper[2579]: E0912 02:04:57.950436 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1780f39c02963fa9995444db819318ee593c09d6c48a1b87258038f9665d0d3\": container with ID starting with c1780f39c02963fa9995444db819318ee593c09d6c48a1b87258038f9665d0d3 not found: ID does not exist" containerID="c1780f39c02963fa9995444db819318ee593c09d6c48a1b87258038f9665d0d3" Sep 12 02:04:57.950606 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:57.950582 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1780f39c02963fa9995444db819318ee593c09d6c48a1b87258038f9665d0d3"} err="failed to get container status \"c1780f39c02963fa9995444db819318ee593c09d6c48a1b87258038f9665d0d3\": rpc error: code = NotFound desc = could not find container \"c1780f39c02963fa9995444db819318ee593c09d6c48a1b87258038f9665d0d3\": container with ID starting with c1780f39c02963fa9995444db819318ee593c09d6c48a1b87258038f9665d0d3 not found: ID does not exist" Sep 12 02:04:58.004683 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:58.004651 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8063-8be879" Sep 12 02:04:58.283778 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:58.283754 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8063-8be879"] Sep 12 02:04:58.285888 ip-10-0-133-79 kubenswrapper[2579]: W0912 02:04:58.285858 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f6afdca_e7c6_4bfc_9ffe_9acd05d11e0c.slice/crio-948a61a2c7e07029858ba2f78750a1f61dc8eb36ef69bf023d0539a6725acb42 WatchSource:0}: Error finding container 948a61a2c7e07029858ba2f78750a1f61dc8eb36ef69bf023d0539a6725acb42: Status 404 returned error can't find the container with id 948a61a2c7e07029858ba2f78750a1f61dc8eb36ef69bf023d0539a6725acb42 Sep 12 02:04:58.657762 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:58.657710 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:04:58.942415 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:58.942384 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f660a6/distinct-2-w0-f660a6-kserve-5d76f79fd5-ksmq5"] Sep 12 02:04:58.948673 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:58.948652 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f660a6/distinct-2-w0-f660a6-kserve-5d76f79fd5-ksmq5" Sep 12 02:04:58.949382 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:58.949354 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8063-8be879" event={"ID":"0f6afdca-e7c6-4bfc-9ffe-9acd05d11e0c","Type":"ContainerStarted","Data":"d30c021941d6f8266823d48ec31abd883ba2b3f7b5721c00ca061445f6e5f21b"} Sep 12 02:04:58.949527 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:58.949390 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8063-8be879" event={"ID":"0f6afdca-e7c6-4bfc-9ffe-9acd05d11e0c","Type":"ContainerStarted","Data":"948a61a2c7e07029858ba2f78750a1f61dc8eb36ef69bf023d0539a6725acb42"} Sep 12 02:04:58.950621 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:58.950441 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8063-be83c6" podUID="33168515-4d1d-4972-a813-1e475ef15769" containerName="test-curl-8063-be83c6" containerID="cri-o://910ab83d6a828b94415afff6684b80bfe0ef5f11c238ac58ae194b86a4858400" gracePeriod=2 Sep 12 02:04:58.952119 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:58.951930 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f660a6\"/\"distinct-2-w0-f660a6-kserve-self-signed-certs\"" Sep 12 02:04:58.954679 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:58.954655 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f660a6/distinct-2-w0-f660a6-kserve-5d76f79fd5-ksmq5"] Sep 12 02:04:58.977581 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:58.977521 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8063-8be879" podStartSLOduration=1.977498113 podStartE2EDuration="1.977498113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:04:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:04:58.976924845 +0000 UTC m=+1201.768918642" watchObservedRunningTime="2026-09-12 02:04:58.977498113 +0000 UTC m=+1201.769491909" Sep 12 02:04:58.995621 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:58.995590 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c25279e-6e3a-4104-9934-a6707fa63885-home\") pod \"distinct-2-w0-f660a6-kserve-5d76f79fd5-ksmq5\" (UID: \"6c25279e-6e3a-4104-9934-a6707fa63885\") " pod="e2e-models-e2e-worker-w0-f660a6/distinct-2-w0-f660a6-kserve-5d76f79fd5-ksmq5" Sep 12 02:04:58.995751 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:58.995632 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c25279e-6e3a-4104-9934-a6707fa63885-tmp-dir\") pod \"distinct-2-w0-f660a6-kserve-5d76f79fd5-ksmq5\" (UID: \"6c25279e-6e3a-4104-9934-a6707fa63885\") " pod="e2e-models-e2e-worker-w0-f660a6/distinct-2-w0-f660a6-kserve-5d76f79fd5-ksmq5" Sep 12 02:04:58.995819 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:58.995801 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c25279e-6e3a-4104-9934-a6707fa63885-model-cache\") pod \"distinct-2-w0-f660a6-kserve-5d76f79fd5-ksmq5\" (UID: \"6c25279e-6e3a-4104-9934-a6707fa63885\") " pod="e2e-models-e2e-worker-w0-f660a6/distinct-2-w0-f660a6-kserve-5d76f79fd5-ksmq5" Sep 12 02:04:58.995878 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:58.995838 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c4ng\" (UniqueName: \"kubernetes.io/projected/6c25279e-6e3a-4104-9934-a6707fa63885-kube-api-access-9c4ng\") pod \"distinct-2-w0-f660a6-kserve-5d76f79fd5-ksmq5\" (UID: \"6c25279e-6e3a-4104-9934-a6707fa63885\") " pod="e2e-models-e2e-worker-w0-f660a6/distinct-2-w0-f660a6-kserve-5d76f79fd5-ksmq5" Sep 12 02:04:58.995878 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:58.995868 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c25279e-6e3a-4104-9934-a6707fa63885-dshm\") pod \"distinct-2-w0-f660a6-kserve-5d76f79fd5-ksmq5\" (UID: \"6c25279e-6e3a-4104-9934-a6707fa63885\") " pod="e2e-models-e2e-worker-w0-f660a6/distinct-2-w0-f660a6-kserve-5d76f79fd5-ksmq5" Sep 12 02:04:58.995991 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:58.995967 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c25279e-6e3a-4104-9934-a6707fa63885-tls-certs\") pod \"distinct-2-w0-f660a6-kserve-5d76f79fd5-ksmq5\" (UID: \"6c25279e-6e3a-4104-9934-a6707fa63885\") " pod="e2e-models-e2e-worker-w0-f660a6/distinct-2-w0-f660a6-kserve-5d76f79fd5-ksmq5" Sep 12 02:04:59.097574 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:59.097535 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c25279e-6e3a-4104-9934-a6707fa63885-home\") pod \"distinct-2-w0-f660a6-kserve-5d76f79fd5-ksmq5\" (UID: \"6c25279e-6e3a-4104-9934-a6707fa63885\") " pod="e2e-models-e2e-worker-w0-f660a6/distinct-2-w0-f660a6-kserve-5d76f79fd5-ksmq5" Sep 12 02:04:59.097749 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:59.097594 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c25279e-6e3a-4104-9934-a6707fa63885-tmp-dir\") pod \"distinct-2-w0-f660a6-kserve-5d76f79fd5-ksmq5\" (UID: \"6c25279e-6e3a-4104-9934-a6707fa63885\") " pod="e2e-models-e2e-worker-w0-f660a6/distinct-2-w0-f660a6-kserve-5d76f79fd5-ksmq5" Sep 12 02:04:59.097749 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:59.097675 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c25279e-6e3a-4104-9934-a6707fa63885-model-cache\") pod \"distinct-2-w0-f660a6-kserve-5d76f79fd5-ksmq5\" (UID: \"6c25279e-6e3a-4104-9934-a6707fa63885\") " pod="e2e-models-e2e-worker-w0-f660a6/distinct-2-w0-f660a6-kserve-5d76f79fd5-ksmq5" Sep 12 02:04:59.097749 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:59.097709 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c4ng\" (UniqueName: \"kubernetes.io/projected/6c25279e-6e3a-4104-9934-a6707fa63885-kube-api-access-9c4ng\") pod \"distinct-2-w0-f660a6-kserve-5d76f79fd5-ksmq5\" (UID: \"6c25279e-6e3a-4104-9934-a6707fa63885\") " pod="e2e-models-e2e-worker-w0-f660a6/distinct-2-w0-f660a6-kserve-5d76f79fd5-ksmq5" Sep 12 02:04:59.097749 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:59.097739 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c25279e-6e3a-4104-9934-a6707fa63885-dshm\") pod \"distinct-2-w0-f660a6-kserve-5d76f79fd5-ksmq5\" (UID: \"6c25279e-6e3a-4104-9934-a6707fa63885\") " pod="e2e-models-e2e-worker-w0-f660a6/distinct-2-w0-f660a6-kserve-5d76f79fd5-ksmq5" Sep 12 02:04:59.098003 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:59.097794 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c25279e-6e3a-4104-9934-a6707fa63885-tls-certs\") pod \"distinct-2-w0-f660a6-kserve-5d76f79fd5-ksmq5\" (UID: \"6c25279e-6e3a-4104-9934-a6707fa63885\") " pod="e2e-models-e2e-worker-w0-f660a6/distinct-2-w0-f660a6-kserve-5d76f79fd5-ksmq5" Sep 12 02:04:59.098003 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:59.097966 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c25279e-6e3a-4104-9934-a6707fa63885-home\") pod \"distinct-2-w0-f660a6-kserve-5d76f79fd5-ksmq5\" (UID: \"6c25279e-6e3a-4104-9934-a6707fa63885\") " pod="e2e-models-e2e-worker-w0-f660a6/distinct-2-w0-f660a6-kserve-5d76f79fd5-ksmq5" Sep 12 02:04:59.098257 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:59.098236 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c25279e-6e3a-4104-9934-a6707fa63885-tmp-dir\") pod \"distinct-2-w0-f660a6-kserve-5d76f79fd5-ksmq5\" (UID: \"6c25279e-6e3a-4104-9934-a6707fa63885\") " pod="e2e-models-e2e-worker-w0-f660a6/distinct-2-w0-f660a6-kserve-5d76f79fd5-ksmq5" Sep 12 02:04:59.098596 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:59.098435 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c25279e-6e3a-4104-9934-a6707fa63885-model-cache\") pod \"distinct-2-w0-f660a6-kserve-5d76f79fd5-ksmq5\" (UID: \"6c25279e-6e3a-4104-9934-a6707fa63885\") " pod="e2e-models-e2e-worker-w0-f660a6/distinct-2-w0-f660a6-kserve-5d76f79fd5-ksmq5" Sep 12 02:04:59.101162 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:59.100887 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c25279e-6e3a-4104-9934-a6707fa63885-dshm\") pod \"distinct-2-w0-f660a6-kserve-5d76f79fd5-ksmq5\" (UID: \"6c25279e-6e3a-4104-9934-a6707fa63885\") " pod="e2e-models-e2e-worker-w0-f660a6/distinct-2-w0-f660a6-kserve-5d76f79fd5-ksmq5" Sep 12 02:04:59.101162 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:59.101119 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c25279e-6e3a-4104-9934-a6707fa63885-tls-certs\") pod \"distinct-2-w0-f660a6-kserve-5d76f79fd5-ksmq5\" (UID: \"6c25279e-6e3a-4104-9934-a6707fa63885\") " pod="e2e-models-e2e-worker-w0-f660a6/distinct-2-w0-f660a6-kserve-5d76f79fd5-ksmq5" Sep 12 02:04:59.106361 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:59.106336 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c4ng\" (UniqueName: \"kubernetes.io/projected/6c25279e-6e3a-4104-9934-a6707fa63885-kube-api-access-9c4ng\") pod \"distinct-2-w0-f660a6-kserve-5d76f79fd5-ksmq5\" (UID: \"6c25279e-6e3a-4104-9934-a6707fa63885\") " pod="e2e-models-e2e-worker-w0-f660a6/distinct-2-w0-f660a6-kserve-5d76f79fd5-ksmq5" Sep 12 02:04:59.127896 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:59.127862 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:04:59.133552 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:59.133534 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8063-7a2a8d" Sep 12 02:04:59.199147 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:59.199051 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nngts\" (UniqueName: \"kubernetes.io/projected/aef019f4-965e-4995-8f32-c53741b3157c-kube-api-access-nngts\") pod \"aef019f4-965e-4995-8f32-c53741b3157c\" (UID: \"aef019f4-965e-4995-8f32-c53741b3157c\") " Sep 12 02:04:59.201279 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:59.201242 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aef019f4-965e-4995-8f32-c53741b3157c-kube-api-access-nngts" (OuterVolumeSpecName: "kube-api-access-nngts") pod "aef019f4-965e-4995-8f32-c53741b3157c" (UID: "aef019f4-965e-4995-8f32-c53741b3157c"). InnerVolumeSpecName "kube-api-access-nngts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 12 02:04:59.299929 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:59.299887 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nngts\" (UniqueName: \"kubernetes.io/projected/aef019f4-965e-4995-8f32-c53741b3157c-kube-api-access-nngts\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:04:59.303689 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:59.303668 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f660a6/distinct-2-w0-f660a6-kserve-5d76f79fd5-ksmq5" Sep 12 02:04:59.353233 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:59.353192 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:04:59.578342 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:59.578314 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f660a6/distinct-2-w0-f660a6-kserve-5d76f79fd5-ksmq5"] Sep 12 02:04:59.580815 ip-10-0-133-79 kubenswrapper[2579]: W0912 02:04:59.580785 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c25279e_6e3a_4104_9934_a6707fa63885.slice/crio-ae89eeeb61eb6f131d6fdce87ea22b7a05506161e55060bdb59a428d151c03b8 WatchSource:0}: Error finding container ae89eeeb61eb6f131d6fdce87ea22b7a05506161e55060bdb59a428d151c03b8: Status 404 returned error can't find the container with id ae89eeeb61eb6f131d6fdce87ea22b7a05506161e55060bdb59a428d151c03b8 Sep 12 02:04:59.616840 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:59.616809 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8063-8be879"] Sep 12 02:04:59.620633 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:59.620602 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8063-8be879"] Sep 12 02:04:59.727432 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:59.727401 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aef019f4-965e-4995-8f32-c53741b3157c" path="/var/lib/kubelet/pods/aef019f4-965e-4995-8f32-c53741b3157c/volumes" Sep 12 02:04:59.957316 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:59.957271 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f660a6/distinct-2-w0-f660a6-kserve-5d76f79fd5-ksmq5" event={"ID":"6c25279e-6e3a-4104-9934-a6707fa63885","Type":"ContainerStarted","Data":"70a7f07cf38054d2c8d71084ab5889d4d58ca26d4cc7efdbd8537986faf3727d"} Sep 12 02:04:59.957756 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:59.957327 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f660a6/distinct-2-w0-f660a6-kserve-5d76f79fd5-ksmq5" event={"ID":"6c25279e-6e3a-4104-9934-a6707fa63885","Type":"ContainerStarted","Data":"ae89eeeb61eb6f131d6fdce87ea22b7a05506161e55060bdb59a428d151c03b8"} Sep 12 02:04:59.958555 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:59.958532 2579 generic.go:356] "Generic (PLEG): container finished" podID="aef019f4-965e-4995-8f32-c53741b3157c" containerID="67dbfa77b78ab30af926af5b5ed017959f9909fe08dbaaaa3608b673876ddf4e" exitCode=137 Sep 12 02:04:59.958650 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:59.958570 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8063-7a2a8d" Sep 12 02:04:59.958650 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:59.958635 2579 scope.go:117] "RemoveContainer" containerID="67dbfa77b78ab30af926af5b5ed017959f9909fe08dbaaaa3608b673876ddf4e" Sep 12 02:04:59.958845 ip-10-0-133-79 kubenswrapper[2579]: E0912 02:04:59.958824 2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8063-8be879" container="test-curl-8063-8be879" Sep 12 02:04:59.967681 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:59.967660 2579 scope.go:117] "RemoveContainer" containerID="67dbfa77b78ab30af926af5b5ed017959f9909fe08dbaaaa3608b673876ddf4e" Sep 12 02:04:59.967989 ip-10-0-133-79 kubenswrapper[2579]: E0912 02:04:59.967971 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67dbfa77b78ab30af926af5b5ed017959f9909fe08dbaaaa3608b673876ddf4e\": container with ID starting with 67dbfa77b78ab30af926af5b5ed017959f9909fe08dbaaaa3608b673876ddf4e not found: ID does not exist" containerID="67dbfa77b78ab30af926af5b5ed017959f9909fe08dbaaaa3608b673876ddf4e" Sep 12 02:04:59.968039 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:59.967999 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67dbfa77b78ab30af926af5b5ed017959f9909fe08dbaaaa3608b673876ddf4e"} err="failed to get container status \"67dbfa77b78ab30af926af5b5ed017959f9909fe08dbaaaa3608b673876ddf4e\": rpc error: code = NotFound desc = could not find container \"67dbfa77b78ab30af926af5b5ed017959f9909fe08dbaaaa3608b673876ddf4e\": container with ID starting with 67dbfa77b78ab30af926af5b5ed017959f9909fe08dbaaaa3608b673876ddf4e not found: ID does not exist" Sep 12 02:04:59.975285 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:04:59.975233 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f660a6/distinct-2-w0-f660a6-kserve-5d76f79fd5-ksmq5" podStartSLOduration=1.975216058 podStartE2EDuration="1.975216058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:04:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:04:59.972870805 +0000 UTC m=+1202.764864602" watchObservedRunningTime="2026-09-12 02:04:59.975216058 +0000 UTC m=+1202.767209896" Sep 12 02:05:00.156323 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:00.156286 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:00.256497 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:00.256375 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:00.550044 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:00.549940 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:00.800744 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:00.800664 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:00.966399 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:00.966360 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8063-8be879" podUID="0f6afdca-e7c6-4bfc-9ffe-9acd05d11e0c" containerName="test-curl-8063-8be879" containerID="cri-o://d30c021941d6f8266823d48ec31abd883ba2b3f7b5721c00ca061445f6e5f21b" gracePeriod=2 Sep 12 02:05:00.976823 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:00.976786 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:01.136086 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:01.136061 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8063-be83c6" Sep 12 02:05:01.218526 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:01.218446 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s9rqx\" (UniqueName: \"kubernetes.io/projected/33168515-4d1d-4972-a813-1e475ef15769-kube-api-access-s9rqx\") pod \"33168515-4d1d-4972-a813-1e475ef15769\" (UID: \"33168515-4d1d-4972-a813-1e475ef15769\") " Sep 12 02:05:01.220735 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:01.220709 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33168515-4d1d-4972-a813-1e475ef15769-kube-api-access-s9rqx" (OuterVolumeSpecName: "kube-api-access-s9rqx") pod "33168515-4d1d-4972-a813-1e475ef15769" (UID: "33168515-4d1d-4972-a813-1e475ef15769"). InnerVolumeSpecName "kube-api-access-s9rqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 12 02:05:01.320244 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:01.320147 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9rqx\" (UniqueName: \"kubernetes.io/projected/33168515-4d1d-4972-a813-1e475ef15769-kube-api-access-s9rqx\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:05:01.727141 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:01.727107 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33168515-4d1d-4972-a813-1e475ef15769" path="/var/lib/kubelet/pods/33168515-4d1d-4972-a813-1e475ef15769/volumes" Sep 12 02:05:01.847903 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:01.847860 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:01.972834 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:01.972799 2579 generic.go:356] "Generic (PLEG): container finished" podID="33168515-4d1d-4972-a813-1e475ef15769" containerID="910ab83d6a828b94415afff6684b80bfe0ef5f11c238ac58ae194b86a4858400" exitCode=137 Sep 12 02:05:01.973308 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:01.972862 2579 scope.go:117] "RemoveContainer" containerID="910ab83d6a828b94415afff6684b80bfe0ef5f11c238ac58ae194b86a4858400" Sep 12 02:05:01.973308 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:01.972864 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8063-be83c6" Sep 12 02:05:01.982372 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:01.982107 2579 scope.go:117] "RemoveContainer" containerID="910ab83d6a828b94415afff6684b80bfe0ef5f11c238ac58ae194b86a4858400" Sep 12 02:05:01.982598 ip-10-0-133-79 kubenswrapper[2579]: E0912 02:05:01.982578 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"910ab83d6a828b94415afff6684b80bfe0ef5f11c238ac58ae194b86a4858400\": container with ID starting with 910ab83d6a828b94415afff6684b80bfe0ef5f11c238ac58ae194b86a4858400 not found: ID does not exist" containerID="910ab83d6a828b94415afff6684b80bfe0ef5f11c238ac58ae194b86a4858400" Sep 12 02:05:01.982649 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:01.982609 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"910ab83d6a828b94415afff6684b80bfe0ef5f11c238ac58ae194b86a4858400"} err="failed to get container status \"910ab83d6a828b94415afff6684b80bfe0ef5f11c238ac58ae194b86a4858400\": rpc error: code = NotFound desc = could not find container \"910ab83d6a828b94415afff6684b80bfe0ef5f11c238ac58ae194b86a4858400\": container with ID starting with 910ab83d6a828b94415afff6684b80bfe0ef5f11c238ac58ae194b86a4858400 not found: ID does not exist" Sep 12 02:05:02.637044 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:02.637008 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:03.165417 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:03.165390 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8063-8be879" Sep 12 02:05:03.240480 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:03.240379 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w4t69\" (UniqueName: \"kubernetes.io/projected/0f6afdca-e7c6-4bfc-9ffe-9acd05d11e0c-kube-api-access-w4t69\") pod \"0f6afdca-e7c6-4bfc-9ffe-9acd05d11e0c\" (UID: \"0f6afdca-e7c6-4bfc-9ffe-9acd05d11e0c\") " Sep 12 02:05:03.242681 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:03.242652 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f6afdca-e7c6-4bfc-9ffe-9acd05d11e0c-kube-api-access-w4t69" (OuterVolumeSpecName: "kube-api-access-w4t69") pod "0f6afdca-e7c6-4bfc-9ffe-9acd05d11e0c" (UID: "0f6afdca-e7c6-4bfc-9ffe-9acd05d11e0c"). InnerVolumeSpecName "kube-api-access-w4t69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 12 02:05:03.342259 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:03.342221 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4t69\" (UniqueName: \"kubernetes.io/projected/0f6afdca-e7c6-4bfc-9ffe-9acd05d11e0c-kube-api-access-w4t69\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:05:03.460411 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:03.460372 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:03.728662 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:03.728626 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f6afdca-e7c6-4bfc-9ffe-9acd05d11e0c" path="/var/lib/kubelet/pods/0f6afdca-e7c6-4bfc-9ffe-9acd05d11e0c/volumes" Sep 12 02:05:03.905141 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:03.905109 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:03.986418 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:03.986323 2579 generic.go:356] "Generic (PLEG): container finished" podID="0f6afdca-e7c6-4bfc-9ffe-9acd05d11e0c" containerID="d30c021941d6f8266823d48ec31abd883ba2b3f7b5721c00ca061445f6e5f21b" exitCode=137 Sep 12 02:05:03.986418 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:03.986369 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8063-8be879" Sep 12 02:05:03.986418 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:03.986415 2579 scope.go:117] "RemoveContainer" containerID="d30c021941d6f8266823d48ec31abd883ba2b3f7b5721c00ca061445f6e5f21b" Sep 12 02:05:03.995744 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:03.995718 2579 scope.go:117] "RemoveContainer" containerID="d30c021941d6f8266823d48ec31abd883ba2b3f7b5721c00ca061445f6e5f21b" Sep 12 02:05:03.996088 ip-10-0-133-79 kubenswrapper[2579]: E0912 02:05:03.996071 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d30c021941d6f8266823d48ec31abd883ba2b3f7b5721c00ca061445f6e5f21b\": container with ID starting with d30c021941d6f8266823d48ec31abd883ba2b3f7b5721c00ca061445f6e5f21b not found: ID does not exist" containerID="d30c021941d6f8266823d48ec31abd883ba2b3f7b5721c00ca061445f6e5f21b" Sep 12 02:05:03.996144 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:03.996097 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d30c021941d6f8266823d48ec31abd883ba2b3f7b5721c00ca061445f6e5f21b"} err="failed to get container status \"d30c021941d6f8266823d48ec31abd883ba2b3f7b5721c00ca061445f6e5f21b\": rpc error: code = NotFound desc = could not find container \"d30c021941d6f8266823d48ec31abd883ba2b3f7b5721c00ca061445f6e5f21b\": container with ID starting with d30c021941d6f8266823d48ec31abd883ba2b3f7b5721c00ca061445f6e5f21b not found: ID does not exist" Sep 12 02:05:04.005997 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:04.005961 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:04.380864 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:04.380833 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:04.505624 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:04.505593 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-7d6e9f/test-model-7d6e9f-kserve-8c9487947-5pxmt"] Sep 12 02:05:04.525740 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:04.525709 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:05.235724 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:05.235685 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:05.883191 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:05.883140 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:06.632112 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:06.632081 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:06.734578 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:06.734539 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:07.450574 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:07.450540 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:08.053472 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:08.053426 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-60c2473b-7cfb747df6-flgvv"] Sep 12 02:05:08.054038 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:08.054017 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f6afdca-e7c6-4bfc-9ffe-9acd05d11e0c" containerName="test-curl-8063-8be879" Sep 12 02:05:08.054038 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:08.054038 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f6afdca-e7c6-4bfc-9ffe-9acd05d11e0c" containerName="test-curl-8063-8be879" Sep 12 02:05:08.054241 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:08.054135 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f6afdca-e7c6-4bfc-9ffe-9acd05d11e0c" containerName="test-curl-8063-8be879" Sep 12 02:05:08.059582 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:08.059557 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-60c2473b-7cfb747df6-flgvv" Sep 12 02:05:08.061889 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:08.061868 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-60c2473b-serving-cert\"" Sep 12 02:05:08.068356 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:08.068329 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-60c2473b-7cfb747df6-flgvv"] Sep 12 02:05:08.092858 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:08.092823 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:08.192439 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:08.192403 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4adfbf96-ff58-48aa-9604-b6d4463655e2-metrics-certs\") pod \"maas-api-e2e-mt-60c2473b-7cfb747df6-flgvv\" (UID: \"4adfbf96-ff58-48aa-9604-b6d4463655e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-60c2473b-7cfb747df6-flgvv" Sep 12 02:05:08.192636 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:08.192538 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbs7q\" (UniqueName: \"kubernetes.io/projected/4adfbf96-ff58-48aa-9604-b6d4463655e2-kube-api-access-rbs7q\") pod \"maas-api-e2e-mt-60c2473b-7cfb747df6-flgvv\" (UID: \"4adfbf96-ff58-48aa-9604-b6d4463655e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-60c2473b-7cfb747df6-flgvv" Sep 12 02:05:08.192636 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:08.192562 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4adfbf96-ff58-48aa-9604-b6d4463655e2-openshift-service-ca\") pod \"maas-api-e2e-mt-60c2473b-7cfb747df6-flgvv\" (UID: \"4adfbf96-ff58-48aa-9604-b6d4463655e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-60c2473b-7cfb747df6-flgvv" Sep 12 02:05:08.192636 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:08.192585 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4adfbf96-ff58-48aa-9604-b6d4463655e2-maas-api-tls\") pod \"maas-api-e2e-mt-60c2473b-7cfb747df6-flgvv\" (UID: \"4adfbf96-ff58-48aa-9604-b6d4463655e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-60c2473b-7cfb747df6-flgvv" Sep 12 02:05:08.194307 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:08.194276 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:08.293364 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:08.293332 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4adfbf96-ff58-48aa-9604-b6d4463655e2-metrics-certs\") pod \"maas-api-e2e-mt-60c2473b-7cfb747df6-flgvv\" (UID: \"4adfbf96-ff58-48aa-9604-b6d4463655e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-60c2473b-7cfb747df6-flgvv" Sep 12 02:05:08.293583 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:08.293488 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbs7q\" (UniqueName: \"kubernetes.io/projected/4adfbf96-ff58-48aa-9604-b6d4463655e2-kube-api-access-rbs7q\") pod \"maas-api-e2e-mt-60c2473b-7cfb747df6-flgvv\" (UID: \"4adfbf96-ff58-48aa-9604-b6d4463655e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-60c2473b-7cfb747df6-flgvv" Sep 12 02:05:08.293583 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:08.293523 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4adfbf96-ff58-48aa-9604-b6d4463655e2-openshift-service-ca\") pod \"maas-api-e2e-mt-60c2473b-7cfb747df6-flgvv\" (UID: \"4adfbf96-ff58-48aa-9604-b6d4463655e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-60c2473b-7cfb747df6-flgvv" Sep 12 02:05:08.293583 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:08.293556 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4adfbf96-ff58-48aa-9604-b6d4463655e2-maas-api-tls\") pod \"maas-api-e2e-mt-60c2473b-7cfb747df6-flgvv\" (UID: \"4adfbf96-ff58-48aa-9604-b6d4463655e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-60c2473b-7cfb747df6-flgvv" Sep 12 02:05:08.293767 ip-10-0-133-79 kubenswrapper[2579]: E0912 02:05:08.293712 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-60c2473b-serving-cert: secret "maas-api-e2e-mt-60c2473b-serving-cert" not found Sep 12 02:05:08.293813 ip-10-0-133-79 kubenswrapper[2579]: E0912 02:05:08.293795 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4adfbf96-ff58-48aa-9604-b6d4463655e2-maas-api-tls podName:4adfbf96-ff58-48aa-9604-b6d4463655e2 nodeName:}" failed. No retries permitted until 2026-09-12 02:05:08.793776448 +0000 UTC m=+1211.585770237 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4adfbf96-ff58-48aa-9604-b6d4463655e2-maas-api-tls") pod "maas-api-e2e-mt-60c2473b-7cfb747df6-flgvv" (UID: "4adfbf96-ff58-48aa-9604-b6d4463655e2") : secret "maas-api-e2e-mt-60c2473b-serving-cert" not found Sep 12 02:05:08.294210 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:08.294188 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4adfbf96-ff58-48aa-9604-b6d4463655e2-openshift-service-ca\") pod \"maas-api-e2e-mt-60c2473b-7cfb747df6-flgvv\" (UID: \"4adfbf96-ff58-48aa-9604-b6d4463655e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-60c2473b-7cfb747df6-flgvv" Sep 12 02:05:08.296026 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:08.296004 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4adfbf96-ff58-48aa-9604-b6d4463655e2-metrics-certs\") pod \"maas-api-e2e-mt-60c2473b-7cfb747df6-flgvv\" (UID: \"4adfbf96-ff58-48aa-9604-b6d4463655e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-60c2473b-7cfb747df6-flgvv" Sep 12 02:05:08.304244 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:08.304172 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbs7q\" (UniqueName: \"kubernetes.io/projected/4adfbf96-ff58-48aa-9604-b6d4463655e2-kube-api-access-rbs7q\") pod \"maas-api-e2e-mt-60c2473b-7cfb747df6-flgvv\" (UID: \"4adfbf96-ff58-48aa-9604-b6d4463655e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-60c2473b-7cfb747df6-flgvv" Sep 12 02:05:08.576472 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:08.576366 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:08.644172 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:08.644135 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:08.651342 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:08.651314 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-60c2473b-5b4698ff58-nhcs4"] Sep 12 02:05:08.655519 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:08.655502 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-60c2473b-5b4698ff58-nhcs4" Sep 12 02:05:08.658247 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:08.658215 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-60c2473b-dockercfg-jdgsb\"" Sep 12 02:05:08.658367 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:08.658255 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-60c2473b\"" Sep 12 02:05:08.664853 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:08.664830 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-60c2473b-5b4698ff58-nhcs4"] Sep 12 02:05:08.798929 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:08.798897 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4a62cf7-0de1-46b2-9242-0f0d66fca18d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-60c2473b-5b4698ff58-nhcs4\" (UID: \"c4a62cf7-0de1-46b2-9242-0f0d66fca18d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-60c2473b-5b4698ff58-nhcs4" Sep 12 02:05:08.798929 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:08.798932 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcdb8\" (UniqueName: \"kubernetes.io/projected/c4a62cf7-0de1-46b2-9242-0f0d66fca18d-kube-api-access-rcdb8\") pod \"payload-pre-processing-e2e-mt-60c2473b-5b4698ff58-nhcs4\" (UID: \"c4a62cf7-0de1-46b2-9242-0f0d66fca18d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-60c2473b-5b4698ff58-nhcs4" Sep 12 02:05:08.799142 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:08.799064 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4adfbf96-ff58-48aa-9604-b6d4463655e2-maas-api-tls\") pod \"maas-api-e2e-mt-60c2473b-7cfb747df6-flgvv\" (UID: \"4adfbf96-ff58-48aa-9604-b6d4463655e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-60c2473b-7cfb747df6-flgvv" Sep 12 02:05:08.801607 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:08.801582 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4adfbf96-ff58-48aa-9604-b6d4463655e2-maas-api-tls\") pod \"maas-api-e2e-mt-60c2473b-7cfb747df6-flgvv\" (UID: \"4adfbf96-ff58-48aa-9604-b6d4463655e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-60c2473b-7cfb747df6-flgvv" Sep 12 02:05:08.900817 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:08.900778 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4a62cf7-0de1-46b2-9242-0f0d66fca18d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-60c2473b-5b4698ff58-nhcs4\" (UID: \"c4a62cf7-0de1-46b2-9242-0f0d66fca18d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-60c2473b-5b4698ff58-nhcs4" Sep 12 02:05:08.900985 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:08.900825 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcdb8\" (UniqueName: \"kubernetes.io/projected/c4a62cf7-0de1-46b2-9242-0f0d66fca18d-kube-api-access-rcdb8\") pod \"payload-pre-processing-e2e-mt-60c2473b-5b4698ff58-nhcs4\" (UID: \"c4a62cf7-0de1-46b2-9242-0f0d66fca18d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-60c2473b-5b4698ff58-nhcs4" Sep 12 02:05:08.901406 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:08.901388 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4a62cf7-0de1-46b2-9242-0f0d66fca18d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-60c2473b-5b4698ff58-nhcs4\" (UID: \"c4a62cf7-0de1-46b2-9242-0f0d66fca18d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-60c2473b-5b4698ff58-nhcs4" Sep 12 02:05:08.908966 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:08.908932 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcdb8\" (UniqueName: \"kubernetes.io/projected/c4a62cf7-0de1-46b2-9242-0f0d66fca18d-kube-api-access-rcdb8\") pod \"payload-pre-processing-e2e-mt-60c2473b-5b4698ff58-nhcs4\" (UID: \"c4a62cf7-0de1-46b2-9242-0f0d66fca18d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-60c2473b-5b4698ff58-nhcs4" Sep 12 02:05:08.957362 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:08.957325 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:08.969333 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:08.969305 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-60c2473b-5b4698ff58-nhcs4" Sep 12 02:05:08.975885 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:08.975857 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-60c2473b-7cfb747df6-flgvv" Sep 12 02:05:09.246703 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:09.246677 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-60c2473b-5b4698ff58-nhcs4"] Sep 12 02:05:09.247641 ip-10-0-133-79 kubenswrapper[2579]: W0912 02:05:09.247617 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4a62cf7_0de1_46b2_9242_0f0d66fca18d.slice/crio-ff76f770d317992039c3d698bda5591e05e11d5edd480ad98e08240db018117c WatchSource:0}: Error finding container ff76f770d317992039c3d698bda5591e05e11d5edd480ad98e08240db018117c: Status 404 returned error can't find the container with id ff76f770d317992039c3d698bda5591e05e11d5edd480ad98e08240db018117c Sep 12 02:05:09.259221 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:09.259200 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-60c2473b-7cfb747df6-flgvv"] Sep 12 02:05:09.262267 ip-10-0-133-79 kubenswrapper[2579]: W0912 02:05:09.262245 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4adfbf96_ff58_48aa_9604_b6d4463655e2.slice/crio-10c51f37027cd22a3add4999978641568a21c445b649f5d9158d9b881358ddb6 WatchSource:0}: Error finding container 10c51f37027cd22a3add4999978641568a21c445b649f5d9158d9b881358ddb6: Status 404 returned error can't find the container with id 10c51f37027cd22a3add4999978641568a21c445b649f5d9158d9b881358ddb6 Sep 12 02:05:09.304251 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:09.304223 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f660a6/distinct-2-w0-f660a6-kserve-5d76f79fd5-ksmq5" Sep 12 02:05:09.304599 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:09.304570 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f660a6/distinct-2-w0-f660a6-kserve-5d76f79fd5-ksmq5" Sep 12 02:05:09.318889 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:09.318864 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f660a6/distinct-2-w0-f660a6-kserve-5d76f79fd5-ksmq5" Sep 12 02:05:09.652952 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:09.652919 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:10.023823 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:10.023714 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-60c2473b-5b4698ff58-nhcs4" event={"ID":"c4a62cf7-0de1-46b2-9242-0f0d66fca18d","Type":"ContainerStarted","Data":"33d3b1b4e4410f5a7d63eb9e56aac3fbbd1a4317b2f9f8c491355b92e0a23e62"} Sep 12 02:05:10.023823 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:10.023777 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-60c2473b-5b4698ff58-nhcs4" event={"ID":"c4a62cf7-0de1-46b2-9242-0f0d66fca18d","Type":"ContainerStarted","Data":"ff76f770d317992039c3d698bda5591e05e11d5edd480ad98e08240db018117c"} Sep 12 02:05:10.024061 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:10.023922 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-60c2473b-5b4698ff58-nhcs4" Sep 12 02:05:10.025390 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:10.025365 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-60c2473b-7cfb747df6-flgvv" event={"ID":"4adfbf96-ff58-48aa-9604-b6d4463655e2","Type":"ContainerStarted","Data":"b03d221b6267ec7591513bdab12ce287ba8028615066bedf33337309a55fb3c3"} Sep 12 02:05:10.025540 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:10.025411 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-60c2473b-7cfb747df6-flgvv" event={"ID":"4adfbf96-ff58-48aa-9604-b6d4463655e2","Type":"ContainerStarted","Data":"10c51f37027cd22a3add4999978641568a21c445b649f5d9158d9b881358ddb6"} Sep 12 02:05:10.038001 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:10.037976 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f660a6/distinct-2-w0-f660a6-kserve-5d76f79fd5-ksmq5" Sep 12 02:05:10.040878 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:10.040836 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-60c2473b-5b4698ff58-nhcs4" podStartSLOduration=2.040819989 podStartE2EDuration="2.040819989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:05:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:05:10.039982561 +0000 UTC m=+1212.831976357" watchObservedRunningTime="2026-09-12 02:05:10.040819989 +0000 UTC m=+1212.832813783" Sep 12 02:05:10.057032 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:10.056976 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-60c2473b-7cfb747df6-flgvv" podStartSLOduration=1.617245289 podStartE2EDuration="2.056957672s" totalImagesPullingTime="439.712383ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:05:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:05:10.056305333 +0000 UTC m=+1212.848299127" watchObservedRunningTime="2026-09-12 02:05:10.056957672 +0000 UTC m=+1212.848951466" Sep 12 02:05:10.295926 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:10.295826 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:10.323065 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:10.323031 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7d6e9f-758b6b6f8b-hdf5b"] Sep 12 02:05:10.323408 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:10.323320 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7d6e9f-758b6b6f8b-hdf5b" podUID="b3213bb0-34c6-4614-9da9-64119a400de7" containerName="payload-pre-processing" containerID="cri-o://460cf300a373fe6d4c576eaa87b116076752d2e523953a9a707063a9e5c7e539" gracePeriod=30 Sep 12 02:05:10.374287 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:10.374247 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:10.474331 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:10.474291 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:10.574108 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:10.574020 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:10.607584 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:10.607560 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7d6e9f-758b6b6f8b-hdf5b" Sep 12 02:05:10.650822 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:10.650781 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:10.719985 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:10.719953 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3213bb0-34c6-4614-9da9-64119a400de7-plugins-config-volume\") pod \"b3213bb0-34c6-4614-9da9-64119a400de7\" (UID: \"b3213bb0-34c6-4614-9da9-64119a400de7\") " Sep 12 02:05:10.720486 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:10.720020 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hsmwr\" (UniqueName: \"kubernetes.io/projected/b3213bb0-34c6-4614-9da9-64119a400de7-kube-api-access-hsmwr\") pod \"b3213bb0-34c6-4614-9da9-64119a400de7\" (UID: \"b3213bb0-34c6-4614-9da9-64119a400de7\") " Sep 12 02:05:10.720486 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:10.720369 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3213bb0-34c6-4614-9da9-64119a400de7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b3213bb0-34c6-4614-9da9-64119a400de7" (UID: "b3213bb0-34c6-4614-9da9-64119a400de7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 12 02:05:10.720591 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:10.720497 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3213bb0-34c6-4614-9da9-64119a400de7-plugins-config-volume\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:05:10.722303 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:10.722273 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3213bb0-34c6-4614-9da9-64119a400de7-kube-api-access-hsmwr" (OuterVolumeSpecName: "kube-api-access-hsmwr") pod "b3213bb0-34c6-4614-9da9-64119a400de7" (UID: "b3213bb0-34c6-4614-9da9-64119a400de7"). InnerVolumeSpecName "kube-api-access-hsmwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 12 02:05:10.821892 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:10.821861 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsmwr\" (UniqueName: \"kubernetes.io/projected/b3213bb0-34c6-4614-9da9-64119a400de7-kube-api-access-hsmwr\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:05:10.878186 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:10.878157 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:11.033243 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:11.033212 2579 generic.go:356] "Generic (PLEG): container finished" podID="b3213bb0-34c6-4614-9da9-64119a400de7" containerID="460cf300a373fe6d4c576eaa87b116076752d2e523953a9a707063a9e5c7e539" exitCode=0 Sep 12 02:05:11.033416 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:11.033254 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7d6e9f-758b6b6f8b-hdf5b" Sep 12 02:05:11.033416 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:11.033284 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7d6e9f-758b6b6f8b-hdf5b" event={"ID":"b3213bb0-34c6-4614-9da9-64119a400de7","Type":"ContainerDied","Data":"460cf300a373fe6d4c576eaa87b116076752d2e523953a9a707063a9e5c7e539"} Sep 12 02:05:11.033416 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:11.033324 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7d6e9f-758b6b6f8b-hdf5b" event={"ID":"b3213bb0-34c6-4614-9da9-64119a400de7","Type":"ContainerDied","Data":"860ba7f1d276fc094a1afeb4c06becd643949156d389d996188485ded7d028ca"} Sep 12 02:05:11.033416 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:11.033345 2579 scope.go:117] "RemoveContainer" containerID="460cf300a373fe6d4c576eaa87b116076752d2e523953a9a707063a9e5c7e539" Sep 12 02:05:11.033824 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:11.033768 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-60c2473b-7cfb747df6-flgvv" Sep 12 02:05:11.043668 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:11.043647 2579 scope.go:117] "RemoveContainer" containerID="460cf300a373fe6d4c576eaa87b116076752d2e523953a9a707063a9e5c7e539" Sep 12 02:05:11.043994 ip-10-0-133-79 kubenswrapper[2579]: E0912 02:05:11.043969 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"460cf300a373fe6d4c576eaa87b116076752d2e523953a9a707063a9e5c7e539\": container with ID starting with 460cf300a373fe6d4c576eaa87b116076752d2e523953a9a707063a9e5c7e539 not found: ID does not exist" containerID="460cf300a373fe6d4c576eaa87b116076752d2e523953a9a707063a9e5c7e539" Sep 12 02:05:11.044057 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:11.044008 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"460cf300a373fe6d4c576eaa87b116076752d2e523953a9a707063a9e5c7e539"} err="failed to get container status \"460cf300a373fe6d4c576eaa87b116076752d2e523953a9a707063a9e5c7e539\": rpc error: code = NotFound desc = could not find container \"460cf300a373fe6d4c576eaa87b116076752d2e523953a9a707063a9e5c7e539\": container with ID starting with 460cf300a373fe6d4c576eaa87b116076752d2e523953a9a707063a9e5c7e539 not found: ID does not exist" Sep 12 02:05:11.055102 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:11.055075 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7d6e9f-758b6b6f8b-hdf5b"] Sep 12 02:05:11.062991 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:11.062960 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7d6e9f-758b6b6f8b-hdf5b"] Sep 12 02:05:11.359313 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:11.359212 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:11.726808 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:11.726775 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3213bb0-34c6-4614-9da9-64119a400de7" path="/var/lib/kubelet/pods/b3213bb0-34c6-4614-9da9-64119a400de7/volumes" Sep 12 02:05:12.125197 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:12.125159 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:12.776905 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:12.776871 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:13.401503 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:13.401466 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:14.755769 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:14.755730 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:15.954533 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:15.954504 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:17.044783 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:17.044756 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-60c2473b-7cfb747df6-flgvv" Sep 12 02:05:18.674986 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:18.674949 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:05:19.526386 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:19.526313 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-7d6e9f/test-model-7d6e9f-kserve-8c9487947-5pxmt" podUID="6a408541-bb13-47c3-a3ef-238f07e989fa" containerName="main" containerID="cri-o://bd85807b4ee9ce079a0c5a72f586b810e42369e1969b51c77930ecf68c96f636" gracePeriod=45 Sep 12 02:05:19.785560 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:19.785493 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7d6e9f/test-model-7d6e9f-kserve-8c9487947-5pxmt" Sep 12 02:05:19.914946 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:19.914893 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a408541-bb13-47c3-a3ef-238f07e989fa-tls-certs\") pod \"6a408541-bb13-47c3-a3ef-238f07e989fa\" (UID: \"6a408541-bb13-47c3-a3ef-238f07e989fa\") " Sep 12 02:05:19.914946 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:19.914951 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a408541-bb13-47c3-a3ef-238f07e989fa-dshm\") pod \"6a408541-bb13-47c3-a3ef-238f07e989fa\" (UID: \"6a408541-bb13-47c3-a3ef-238f07e989fa\") " Sep 12 02:05:19.915220 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:19.914990 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a408541-bb13-47c3-a3ef-238f07e989fa-model-cache\") pod \"6a408541-bb13-47c3-a3ef-238f07e989fa\" (UID: \"6a408541-bb13-47c3-a3ef-238f07e989fa\") " Sep 12 02:05:19.915220 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:19.915015 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a408541-bb13-47c3-a3ef-238f07e989fa-home\") pod \"6a408541-bb13-47c3-a3ef-238f07e989fa\" (UID: \"6a408541-bb13-47c3-a3ef-238f07e989fa\") " Sep 12 02:05:19.915220 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:19.915043 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mv9lk\" (UniqueName: \"kubernetes.io/projected/6a408541-bb13-47c3-a3ef-238f07e989fa-kube-api-access-mv9lk\") pod \"6a408541-bb13-47c3-a3ef-238f07e989fa\" (UID: \"6a408541-bb13-47c3-a3ef-238f07e989fa\") " Sep 12 02:05:19.915220 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:19.915088 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a408541-bb13-47c3-a3ef-238f07e989fa-tmp-dir\") pod \"6a408541-bb13-47c3-a3ef-238f07e989fa\" (UID: \"6a408541-bb13-47c3-a3ef-238f07e989fa\") " Sep 12 02:05:19.915415 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:19.915279 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a408541-bb13-47c3-a3ef-238f07e989fa-model-cache" (OuterVolumeSpecName: "model-cache") pod "6a408541-bb13-47c3-a3ef-238f07e989fa" (UID: "6a408541-bb13-47c3-a3ef-238f07e989fa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 12 02:05:19.915415 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:19.915292 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a408541-bb13-47c3-a3ef-238f07e989fa-home" (OuterVolumeSpecName: "home") pod "6a408541-bb13-47c3-a3ef-238f07e989fa" (UID: "6a408541-bb13-47c3-a3ef-238f07e989fa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 12 02:05:19.915569 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:19.915416 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a408541-bb13-47c3-a3ef-238f07e989fa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6a408541-bb13-47c3-a3ef-238f07e989fa" (UID: "6a408541-bb13-47c3-a3ef-238f07e989fa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 12 02:05:19.917468 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:19.917431 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a408541-bb13-47c3-a3ef-238f07e989fa-dshm" (OuterVolumeSpecName: "dshm") pod "6a408541-bb13-47c3-a3ef-238f07e989fa" (UID: "6a408541-bb13-47c3-a3ef-238f07e989fa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 12 02:05:19.917468 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:19.917442 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a408541-bb13-47c3-a3ef-238f07e989fa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6a408541-bb13-47c3-a3ef-238f07e989fa" (UID: "6a408541-bb13-47c3-a3ef-238f07e989fa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 12 02:05:19.917643 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:19.917473 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a408541-bb13-47c3-a3ef-238f07e989fa-kube-api-access-mv9lk" (OuterVolumeSpecName: "kube-api-access-mv9lk") pod "6a408541-bb13-47c3-a3ef-238f07e989fa" (UID: "6a408541-bb13-47c3-a3ef-238f07e989fa"). InnerVolumeSpecName "kube-api-access-mv9lk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 12 02:05:20.016049 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:20.016005 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a408541-bb13-47c3-a3ef-238f07e989fa-model-cache\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:05:20.016049 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:20.016040 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a408541-bb13-47c3-a3ef-238f07e989fa-home\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:05:20.016049 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:20.016050 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mv9lk\" (UniqueName: \"kubernetes.io/projected/6a408541-bb13-47c3-a3ef-238f07e989fa-kube-api-access-mv9lk\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:05:20.016049 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:20.016059 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a408541-bb13-47c3-a3ef-238f07e989fa-tmp-dir\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:05:20.016334 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:20.016068 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a408541-bb13-47c3-a3ef-238f07e989fa-tls-certs\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:05:20.016334 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:20.016076 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a408541-bb13-47c3-a3ef-238f07e989fa-dshm\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:05:20.085686 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:20.085596 2579 generic.go:356] "Generic (PLEG): container finished" podID="6a408541-bb13-47c3-a3ef-238f07e989fa" containerID="bd85807b4ee9ce079a0c5a72f586b810e42369e1969b51c77930ecf68c96f636" exitCode=0 Sep 12 02:05:20.085686 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:20.085649 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7d6e9f/test-model-7d6e9f-kserve-8c9487947-5pxmt" Sep 12 02:05:20.085937 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:20.085691 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7d6e9f/test-model-7d6e9f-kserve-8c9487947-5pxmt" event={"ID":"6a408541-bb13-47c3-a3ef-238f07e989fa","Type":"ContainerDied","Data":"bd85807b4ee9ce079a0c5a72f586b810e42369e1969b51c77930ecf68c96f636"} Sep 12 02:05:20.085937 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:20.085737 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7d6e9f/test-model-7d6e9f-kserve-8c9487947-5pxmt" event={"ID":"6a408541-bb13-47c3-a3ef-238f07e989fa","Type":"ContainerDied","Data":"0a996799430c1400d4b708900218ab15e737c2d55cd7af99af40784b934d8717"} Sep 12 02:05:20.085937 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:20.085754 2579 scope.go:117] "RemoveContainer" containerID="bd85807b4ee9ce079a0c5a72f586b810e42369e1969b51c77930ecf68c96f636" Sep 12 02:05:20.095658 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:20.095640 2579 scope.go:117] "RemoveContainer" containerID="bd85807b4ee9ce079a0c5a72f586b810e42369e1969b51c77930ecf68c96f636" Sep 12 02:05:20.095947 ip-10-0-133-79 kubenswrapper[2579]: E0912 02:05:20.095929 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd85807b4ee9ce079a0c5a72f586b810e42369e1969b51c77930ecf68c96f636\": container with ID starting with bd85807b4ee9ce079a0c5a72f586b810e42369e1969b51c77930ecf68c96f636 not found: ID does not exist" containerID="bd85807b4ee9ce079a0c5a72f586b810e42369e1969b51c77930ecf68c96f636" Sep 12 02:05:20.095997 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:20.095957 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd85807b4ee9ce079a0c5a72f586b810e42369e1969b51c77930ecf68c96f636"} err="failed to get container status \"bd85807b4ee9ce079a0c5a72f586b810e42369e1969b51c77930ecf68c96f636\": rpc error: code = NotFound desc = could not find container \"bd85807b4ee9ce079a0c5a72f586b810e42369e1969b51c77930ecf68c96f636\": container with ID starting with bd85807b4ee9ce079a0c5a72f586b810e42369e1969b51c77930ecf68c96f636 not found: ID does not exist" Sep 12 02:05:20.106158 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:20.106124 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-7d6e9f/test-model-7d6e9f-kserve-8c9487947-5pxmt"] Sep 12 02:05:20.109971 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:20.109948 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-7d6e9f/test-model-7d6e9f-kserve-8c9487947-5pxmt"] Sep 12 02:05:21.036402 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:21.036372 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-60c2473b-5b4698ff58-nhcs4" Sep 12 02:05:21.726617 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:21.726580 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a408541-bb13-47c3-a3ef-238f07e989fa" path="/var/lib/kubelet/pods/6a408541-bb13-47c3-a3ef-238f07e989fa/volumes" Sep 12 02:05:21.972427 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:21.972390 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:22.248241 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:22.248205 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:22.843625 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:22.843593 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:23.168813 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:23.168775 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:23.672887 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:23.672855 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:25.871672 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:25.871636 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:26.505770 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:26.505736 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:26.603563 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:26.603517 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:26.718511 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:26.718445 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:27.616979 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:27.616946 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:28.357099 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:28.357050 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:28.921285 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:28.921251 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:29.160998 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:29.160958 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:29.460023 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:29.459988 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:29.843120 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:29.843022 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:30.633314 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:30.633278 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a744d5-69c5c5bd6c-fnbt9"] Sep 12 02:05:30.633825 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:30.633786 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3213bb0-34c6-4614-9da9-64119a400de7" containerName="payload-pre-processing" Sep 12 02:05:30.633825 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:30.633799 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3213bb0-34c6-4614-9da9-64119a400de7" containerName="payload-pre-processing" Sep 12 02:05:30.633825 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:30.633809 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a408541-bb13-47c3-a3ef-238f07e989fa" containerName="main" Sep 12 02:05:30.633825 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:30.633815 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a408541-bb13-47c3-a3ef-238f07e989fa" containerName="main" Sep 12 02:05:30.634076 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:30.633898 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a408541-bb13-47c3-a3ef-238f07e989fa" containerName="main" Sep 12 02:05:30.634076 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:30.633908 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3213bb0-34c6-4614-9da9-64119a400de7" containerName="payload-pre-processing" Sep 12 02:05:30.637252 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:30.637230 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a744d5-69c5c5bd6c-fnbt9" Sep 12 02:05:30.640892 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:30.640870 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-a744d5-serving-cert\"" Sep 12 02:05:30.648200 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:30.648161 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a744d5-69c5c5bd6c-fnbt9"] Sep 12 02:05:30.666805 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:30.666763 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:30.730813 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:30.730776 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b6147b2-1b11-42b3-a824-fe7525170f7d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-a744d5-69c5c5bd6c-fnbt9\" (UID: \"4b6147b2-1b11-42b3-a824-fe7525170f7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a744d5-69c5c5bd6c-fnbt9" Sep 12 02:05:30.731001 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:30.730851 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b6147b2-1b11-42b3-a824-fe7525170f7d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-a744d5-69c5c5bd6c-fnbt9\" (UID: \"4b6147b2-1b11-42b3-a824-fe7525170f7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a744d5-69c5c5bd6c-fnbt9" Sep 12 02:05:30.731001 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:30.730952 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xpdz\" (UniqueName: \"kubernetes.io/projected/4b6147b2-1b11-42b3-a824-fe7525170f7d-kube-api-access-7xpdz\") pod \"maas-api-e2e-shared-a-w3-a744d5-69c5c5bd6c-fnbt9\" (UID: \"4b6147b2-1b11-42b3-a824-fe7525170f7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a744d5-69c5c5bd6c-fnbt9" Sep 12 02:05:30.731125 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:30.731003 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b6147b2-1b11-42b3-a824-fe7525170f7d-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-a744d5-69c5c5bd6c-fnbt9\" (UID: \"4b6147b2-1b11-42b3-a824-fe7525170f7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a744d5-69c5c5bd6c-fnbt9" Sep 12 02:05:30.766616 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:30.766582 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:30.831894 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:30.831856 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b6147b2-1b11-42b3-a824-fe7525170f7d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-a744d5-69c5c5bd6c-fnbt9\" (UID: \"4b6147b2-1b11-42b3-a824-fe7525170f7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a744d5-69c5c5bd6c-fnbt9" Sep 12 02:05:30.832072 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:30.831927 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b6147b2-1b11-42b3-a824-fe7525170f7d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-a744d5-69c5c5bd6c-fnbt9\" (UID: \"4b6147b2-1b11-42b3-a824-fe7525170f7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a744d5-69c5c5bd6c-fnbt9" Sep 12 02:05:30.832072 ip-10-0-133-79 kubenswrapper[2579]: E0912 02:05:30.832010 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a744d5-serving-cert: secret "maas-api-e2e-shared-a-w3-a744d5-serving-cert" not found Sep 12 02:05:30.832072 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:30.832025 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xpdz\" (UniqueName: \"kubernetes.io/projected/4b6147b2-1b11-42b3-a824-fe7525170f7d-kube-api-access-7xpdz\") pod \"maas-api-e2e-shared-a-w3-a744d5-69c5c5bd6c-fnbt9\" (UID: \"4b6147b2-1b11-42b3-a824-fe7525170f7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a744d5-69c5c5bd6c-fnbt9" Sep 12 02:05:30.832213 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:30.832076 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b6147b2-1b11-42b3-a824-fe7525170f7d-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-a744d5-69c5c5bd6c-fnbt9\" (UID: \"4b6147b2-1b11-42b3-a824-fe7525170f7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a744d5-69c5c5bd6c-fnbt9" Sep 12 02:05:30.832213 ip-10-0-133-79 kubenswrapper[2579]: E0912 02:05:30.832093 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b6147b2-1b11-42b3-a824-fe7525170f7d-maas-api-tls podName:4b6147b2-1b11-42b3-a824-fe7525170f7d nodeName:}" failed. No retries permitted until 2026-09-12 02:05:31.332070436 +0000 UTC m=+1234.124064230 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4b6147b2-1b11-42b3-a824-fe7525170f7d-maas-api-tls") pod "maas-api-e2e-shared-a-w3-a744d5-69c5c5bd6c-fnbt9" (UID: "4b6147b2-1b11-42b3-a824-fe7525170f7d") : secret "maas-api-e2e-shared-a-w3-a744d5-serving-cert" not found Sep 12 02:05:30.832662 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:30.832641 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b6147b2-1b11-42b3-a824-fe7525170f7d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-a744d5-69c5c5bd6c-fnbt9\" (UID: \"4b6147b2-1b11-42b3-a824-fe7525170f7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a744d5-69c5c5bd6c-fnbt9" Sep 12 02:05:30.834675 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:30.834654 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b6147b2-1b11-42b3-a824-fe7525170f7d-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-a744d5-69c5c5bd6c-fnbt9\" (UID: \"4b6147b2-1b11-42b3-a824-fe7525170f7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a744d5-69c5c5bd6c-fnbt9" Sep 12 02:05:30.849982 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:30.849953 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xpdz\" (UniqueName: \"kubernetes.io/projected/4b6147b2-1b11-42b3-a824-fe7525170f7d-kube-api-access-7xpdz\") pod \"maas-api-e2e-shared-a-w3-a744d5-69c5c5bd6c-fnbt9\" (UID: \"4b6147b2-1b11-42b3-a824-fe7525170f7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a744d5-69c5c5bd6c-fnbt9" Sep 12 02:05:31.010604 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:31.010510 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:31.217535 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:31.217500 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:31.318327 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:31.318228 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:31.337806 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:31.337770 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b6147b2-1b11-42b3-a824-fe7525170f7d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-a744d5-69c5c5bd6c-fnbt9\" (UID: \"4b6147b2-1b11-42b3-a824-fe7525170f7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a744d5-69c5c5bd6c-fnbt9" Sep 12 02:05:31.340883 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:31.340856 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b6147b2-1b11-42b3-a824-fe7525170f7d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-a744d5-69c5c5bd6c-fnbt9\" (UID: \"4b6147b2-1b11-42b3-a824-fe7525170f7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a744d5-69c5c5bd6c-fnbt9" Sep 12 02:05:31.553517 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:31.553444 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a744d5-69c5c5bd6c-fnbt9" Sep 12 02:05:31.833427 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:31.833400 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a744d5-69c5c5bd6c-fnbt9"] Sep 12 02:05:31.835780 ip-10-0-133-79 kubenswrapper[2579]: W0912 02:05:31.835755 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b6147b2_1b11_42b3_a824_fe7525170f7d.slice/crio-c888817c0f2f9632e873c8cb52defa1cac0fa7a201e3983c8271d9d0638351bb WatchSource:0}: Error finding container c888817c0f2f9632e873c8cb52defa1cac0fa7a201e3983c8271d9d0638351bb: Status 404 returned error can't find the container with id c888817c0f2f9632e873c8cb52defa1cac0fa7a201e3983c8271d9d0638351bb Sep 12 02:05:31.858720 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:31.858636 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:32.166149 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:32.166116 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a744d5-69c5c5bd6c-fnbt9" event={"ID":"4b6147b2-1b11-42b3-a824-fe7525170f7d","Type":"ContainerStarted","Data":"c888817c0f2f9632e873c8cb52defa1cac0fa7a201e3983c8271d9d0638351bb"} Sep 12 02:05:32.608859 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:32.608773 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:33.174080 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:33.174036 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a744d5-69c5c5bd6c-fnbt9" event={"ID":"4b6147b2-1b11-42b3-a824-fe7525170f7d","Type":"ContainerStarted","Data":"9df793718fe1221f483d55014956d02c07bd8bc159795d1c75fa1da2a246520a"} Sep 12 02:05:33.174513 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:33.174100 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a744d5-69c5c5bd6c-fnbt9" Sep 12 02:05:33.193353 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:33.193294 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a744d5-69c5c5bd6c-fnbt9" podStartSLOduration=2.715730792 podStartE2EDuration="3.193275055s" totalImagesPullingTime="477.544263ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:05:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:05:33.190247418 +0000 UTC m=+1235.982241213" watchObservedRunningTime="2026-09-12 02:05:33.193275055 +0000 UTC m=+1235.985268882" Sep 12 02:05:33.450333 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:33.450240 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:34.150193 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:34.150156 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:35.000638 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:35.000603 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:39.185168 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:39.185140 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a744d5-69c5c5bd6c-fnbt9" Sep 12 02:05:51.628345 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:51.628264 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:52.890265 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:52.890226 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:53.658548 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:53.658510 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:54.275775 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:54.275731 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:55.006926 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:55.006895 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:55.676059 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:55.676028 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:56.300791 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:56.300752 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:56.659312 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:56.659268 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:57.012747 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:57.012655 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:57.113410 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:57.113367 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:57.328444 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:57.328322 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:57.463444 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:57.463408 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:59.123875 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:59.123843 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:05:59.948905 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:05:59.948862 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:06:00.225896 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:06:00.225806 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:06:00.324993 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:06:00.324956 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:06:00.777428 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:06:00.777396 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:06:01.559092 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:06:01.559054 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:06:01.830411 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:06:01.830315 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:06:02.328712 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:06:02.328660 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:06:02.992070 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:06:02.992033 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:06:03.651194 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:06:03.651159 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:06:04.343079 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:06:04.343040 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fbb6d9-7cb9c56db4-h2gdc"] Sep 12 02:06:04.349413 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:06:04.349388 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fbb6d9-7cb9c56db4-h2gdc" Sep 12 02:06:04.352130 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:06:04.352009 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-fbb6d9-serving-cert\"" Sep 12 02:06:04.356069 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:06:04.356045 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fbb6d9-7cb9c56db4-h2gdc"] Sep 12 02:06:04.373723 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:06:04.373692 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:06:04.378820 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:06:04.378797 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5qwc\" (UniqueName: \"kubernetes.io/projected/8cc047c2-a082-41ff-849d-391241409035-kube-api-access-g5qwc\") pod \"maas-api-e2e-shared-b-w3-fbb6d9-7cb9c56db4-h2gdc\" (UID: \"8cc047c2-a082-41ff-849d-391241409035\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fbb6d9-7cb9c56db4-h2gdc" Sep 12 02:06:04.378977 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:06:04.378922 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cc047c2-a082-41ff-849d-391241409035-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-fbb6d9-7cb9c56db4-h2gdc\" (UID: \"8cc047c2-a082-41ff-849d-391241409035\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fbb6d9-7cb9c56db4-h2gdc" Sep 12 02:06:04.378977 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:06:04.378964 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cc047c2-a082-41ff-849d-391241409035-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-fbb6d9-7cb9c56db4-h2gdc\" (UID: \"8cc047c2-a082-41ff-849d-391241409035\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fbb6d9-7cb9c56db4-h2gdc" Sep 12 02:06:04.379050 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:06:04.378996 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cc047c2-a082-41ff-849d-391241409035-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-fbb6d9-7cb9c56db4-h2gdc\" (UID: \"8cc047c2-a082-41ff-849d-391241409035\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fbb6d9-7cb9c56db4-h2gdc" Sep 12 02:06:04.477430 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:06:04.477395 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:06:04.479608 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:06:04.479587 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cc047c2-a082-41ff-849d-391241409035-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-fbb6d9-7cb9c56db4-h2gdc\" (UID: \"8cc047c2-a082-41ff-849d-391241409035\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fbb6d9-7cb9c56db4-h2gdc" Sep 12 02:06:04.479684 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:06:04.479625 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cc047c2-a082-41ff-849d-391241409035-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-fbb6d9-7cb9c56db4-h2gdc\" (UID: \"8cc047c2-a082-41ff-849d-391241409035\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fbb6d9-7cb9c56db4-h2gdc" Sep 12 02:06:04.479684 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:06:04.479654 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cc047c2-a082-41ff-849d-391241409035-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-fbb6d9-7cb9c56db4-h2gdc\" (UID: \"8cc047c2-a082-41ff-849d-391241409035\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fbb6d9-7cb9c56db4-h2gdc" Sep 12 02:06:04.479764 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:06:04.479686 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5qwc\" (UniqueName: \"kubernetes.io/projected/8cc047c2-a082-41ff-849d-391241409035-kube-api-access-g5qwc\") pod \"maas-api-e2e-shared-b-w3-fbb6d9-7cb9c56db4-h2gdc\" (UID: \"8cc047c2-a082-41ff-849d-391241409035\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fbb6d9-7cb9c56db4-h2gdc" Sep 12 02:06:04.480345 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:06:04.480322 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cc047c2-a082-41ff-849d-391241409035-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-fbb6d9-7cb9c56db4-h2gdc\" (UID: \"8cc047c2-a082-41ff-849d-391241409035\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fbb6d9-7cb9c56db4-h2gdc" Sep 12 02:06:04.482530 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:06:04.482508 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cc047c2-a082-41ff-849d-391241409035-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-fbb6d9-7cb9c56db4-h2gdc\" (UID: \"8cc047c2-a082-41ff-849d-391241409035\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fbb6d9-7cb9c56db4-h2gdc" Sep 12 02:06:04.482530 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:06:04.482524 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cc047c2-a082-41ff-849d-391241409035-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-fbb6d9-7cb9c56db4-h2gdc\" (UID: \"8cc047c2-a082-41ff-849d-391241409035\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fbb6d9-7cb9c56db4-h2gdc" Sep 12 02:06:04.488727 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:06:04.488701 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5qwc\" (UniqueName: \"kubernetes.io/projected/8cc047c2-a082-41ff-849d-391241409035-kube-api-access-g5qwc\") pod \"maas-api-e2e-shared-b-w3-fbb6d9-7cb9c56db4-h2gdc\" (UID: \"8cc047c2-a082-41ff-849d-391241409035\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fbb6d9-7cb9c56db4-h2gdc" Sep 12 02:06:04.575085 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:06:04.575043 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:06:04.662992 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:06:04.662960 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fbb6d9-7cb9c56db4-h2gdc" Sep 12 02:06:04.943492 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:06:04.943463 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fbb6d9-7cb9c56db4-h2gdc"] Sep 12 02:06:04.945291 ip-10-0-133-79 kubenswrapper[2579]: W0912 02:06:04.945263 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cc047c2_a082_41ff_849d_391241409035.slice/crio-eff42daeb83dcb0f98a97dd85a7ebdbd01a47dfedca5a941316ff2ac99a65ea2 WatchSource:0}: Error finding container eff42daeb83dcb0f98a97dd85a7ebdbd01a47dfedca5a941316ff2ac99a65ea2: Status 404 returned error can't find the container with id eff42daeb83dcb0f98a97dd85a7ebdbd01a47dfedca5a941316ff2ac99a65ea2 Sep 12 02:06:04.958648 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:06:04.958618 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:06:05.056145 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:06:05.056099 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:06:05.360367 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:06:05.360327 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fbb6d9-7cb9c56db4-h2gdc" event={"ID":"8cc047c2-a082-41ff-849d-391241409035","Type":"ContainerStarted","Data":"f26ffe20e267ea71c2c6e3fb5d932c59da1ada488d5503e3deaaa90f98fb2b3e"} Sep 12 02:06:05.360774 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:06:05.360388 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fbb6d9-7cb9c56db4-h2gdc" event={"ID":"8cc047c2-a082-41ff-849d-391241409035","Type":"ContainerStarted","Data":"eff42daeb83dcb0f98a97dd85a7ebdbd01a47dfedca5a941316ff2ac99a65ea2"} Sep 12 02:06:05.360774 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:06:05.360558 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fbb6d9-7cb9c56db4-h2gdc" Sep 12 02:06:05.378359 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:06:05.378307 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fbb6d9-7cb9c56db4-h2gdc" podStartSLOduration=1.058655696 podStartE2EDuration="1.378289354s" totalImagesPullingTime="319.633658ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:06:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:06:05.376760946 +0000 UTC m=+1268.168754741" watchObservedRunningTime="2026-09-12 02:06:05.378289354 +0000 UTC m=+1268.170283186" Sep 12 02:06:11.371898 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:06:11.371868 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fbb6d9-7cb9c56db4-h2gdc" Sep 12 02:06:12.954227 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:06:12.954176 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:06:13.054597 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:06:13.054563 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:06:22.251260 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:06:22.251224 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:06:25.188717 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:06:25.188684 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:06:27.233630 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:06:27.233594 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:06:28.059017 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:06:28.058975 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:06:28.773797 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:06:28.773735 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:06:29.646410 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:06:29.646381 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:06:30.373444 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:06:30.373409 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:06:30.471384 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:06:30.471351 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:06:31.073700 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:06:31.073663 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:06:31.871322 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:06:31.871287 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:06:32.582618 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:06:32.582582 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:06:33.839361 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:06:33.839319 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:06:49.263190 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:06:49.263156 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:06:57.678219 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:06:57.673002 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:07:07.134079 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:07.134047 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:07:10.190238 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:10.190197 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:07:10.291848 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:10.291813 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:07:15.848754 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:15.848716 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:07:16.555799 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:16.555762 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:07:17.390494 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:17.390439 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:07:18.043948 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:18.043910 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:07:18.767813 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:18.767717 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:07:19.417293 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:19.417247 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:07:20.267712 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:20.267673 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:07:21.076717 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:21.076670 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:07:21.751085 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:21.751053 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:07:30.922576 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:30.922540 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:07:31.033689 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:31.033650 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-60c2473b-7cfb747df6-flgvv"] Sep 12 02:07:31.033963 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:31.033933 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-60c2473b-7cfb747df6-flgvv" podUID="4adfbf96-ff58-48aa-9604-b6d4463655e2" containerName="maas-api" containerID="cri-o://b03d221b6267ec7591513bdab12ce287ba8028615066bedf33337309a55fb3c3" gracePeriod=30 Sep 12 02:07:31.061326 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:31.061285 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:07:31.094853 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:31.094819 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-60c2473b-5b4698ff58-nhcs4"] Sep 12 02:07:31.095158 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:31.095046 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-60c2473b-5b4698ff58-nhcs4" podUID="c4a62cf7-0de1-46b2-9242-0f0d66fca18d" containerName="payload-pre-processing" containerID="cri-o://33d3b1b4e4410f5a7d63eb9e56aac3fbbd1a4317b2f9f8c491355b92e0a23e62" gracePeriod=30 Sep 12 02:07:31.125386 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:31.125313 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:07:31.224748 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:31.224720 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:07:31.286466 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:31.286418 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:07:31.349963 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:31.349938 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-60c2473b-7cfb747df6-flgvv" Sep 12 02:07:31.383610 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:31.383585 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-60c2473b-5b4698ff58-nhcs4" Sep 12 02:07:31.503194 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:31.503103 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rcdb8\" (UniqueName: \"kubernetes.io/projected/c4a62cf7-0de1-46b2-9242-0f0d66fca18d-kube-api-access-rcdb8\") pod \"c4a62cf7-0de1-46b2-9242-0f0d66fca18d\" (UID: \"c4a62cf7-0de1-46b2-9242-0f0d66fca18d\") " Sep 12 02:07:31.503343 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:31.503202 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4adfbf96-ff58-48aa-9604-b6d4463655e2-maas-api-tls\") pod \"4adfbf96-ff58-48aa-9604-b6d4463655e2\" (UID: \"4adfbf96-ff58-48aa-9604-b6d4463655e2\") " Sep 12 02:07:31.503343 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:31.503277 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4a62cf7-0de1-46b2-9242-0f0d66fca18d-plugins-config-volume\") pod \"c4a62cf7-0de1-46b2-9242-0f0d66fca18d\" (UID: \"c4a62cf7-0de1-46b2-9242-0f0d66fca18d\") " Sep 12 02:07:31.503343 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:31.503315 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4adfbf96-ff58-48aa-9604-b6d4463655e2-metrics-certs\") pod \"4adfbf96-ff58-48aa-9604-b6d4463655e2\" (UID: \"4adfbf96-ff58-48aa-9604-b6d4463655e2\") " Sep 12 02:07:31.503531 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:31.503351 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rbs7q\" (UniqueName: \"kubernetes.io/projected/4adfbf96-ff58-48aa-9604-b6d4463655e2-kube-api-access-rbs7q\") pod \"4adfbf96-ff58-48aa-9604-b6d4463655e2\" (UID: \"4adfbf96-ff58-48aa-9604-b6d4463655e2\") " Sep 12 02:07:31.503531 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:31.503377 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4adfbf96-ff58-48aa-9604-b6d4463655e2-openshift-service-ca\") pod \"4adfbf96-ff58-48aa-9604-b6d4463655e2\" (UID: \"4adfbf96-ff58-48aa-9604-b6d4463655e2\") " Sep 12 02:07:31.503800 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:31.503772 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4adfbf96-ff58-48aa-9604-b6d4463655e2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4adfbf96-ff58-48aa-9604-b6d4463655e2" (UID: "4adfbf96-ff58-48aa-9604-b6d4463655e2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 12 02:07:31.503800 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:31.503781 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4a62cf7-0de1-46b2-9242-0f0d66fca18d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c4a62cf7-0de1-46b2-9242-0f0d66fca18d" (UID: "c4a62cf7-0de1-46b2-9242-0f0d66fca18d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 12 02:07:31.505610 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:31.505585 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4adfbf96-ff58-48aa-9604-b6d4463655e2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4adfbf96-ff58-48aa-9604-b6d4463655e2" (UID: "4adfbf96-ff58-48aa-9604-b6d4463655e2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 12 02:07:31.505610 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:31.505602 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4adfbf96-ff58-48aa-9604-b6d4463655e2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4adfbf96-ff58-48aa-9604-b6d4463655e2" (UID: "4adfbf96-ff58-48aa-9604-b6d4463655e2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 12 02:07:31.505988 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:31.505968 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4a62cf7-0de1-46b2-9242-0f0d66fca18d-kube-api-access-rcdb8" (OuterVolumeSpecName: "kube-api-access-rcdb8") pod "c4a62cf7-0de1-46b2-9242-0f0d66fca18d" (UID: "c4a62cf7-0de1-46b2-9242-0f0d66fca18d"). InnerVolumeSpecName "kube-api-access-rcdb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 12 02:07:31.506040 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:31.505981 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4adfbf96-ff58-48aa-9604-b6d4463655e2-kube-api-access-rbs7q" (OuterVolumeSpecName: "kube-api-access-rbs7q") pod "4adfbf96-ff58-48aa-9604-b6d4463655e2" (UID: "4adfbf96-ff58-48aa-9604-b6d4463655e2"). InnerVolumeSpecName "kube-api-access-rbs7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 12 02:07:31.551494 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:31.551463 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:07:31.605409 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:31.605366 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4adfbf96-ff58-48aa-9604-b6d4463655e2-maas-api-tls\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:07:31.605409 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:31.605402 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4a62cf7-0de1-46b2-9242-0f0d66fca18d-plugins-config-volume\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:07:31.605409 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:31.605416 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4adfbf96-ff58-48aa-9604-b6d4463655e2-metrics-certs\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:07:31.605695 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:31.605428 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbs7q\" (UniqueName: \"kubernetes.io/projected/4adfbf96-ff58-48aa-9604-b6d4463655e2-kube-api-access-rbs7q\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:07:31.605695 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:31.605444 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4adfbf96-ff58-48aa-9604-b6d4463655e2-openshift-service-ca\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:07:31.605695 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:31.605495 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcdb8\" (UniqueName: \"kubernetes.io/projected/c4a62cf7-0de1-46b2-9242-0f0d66fca18d-kube-api-access-rcdb8\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:07:31.869613 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:31.869517 2579 generic.go:356] "Generic (PLEG): container finished" podID="c4a62cf7-0de1-46b2-9242-0f0d66fca18d" containerID="33d3b1b4e4410f5a7d63eb9e56aac3fbbd1a4317b2f9f8c491355b92e0a23e62" exitCode=0 Sep 12 02:07:31.869613 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:31.869567 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-60c2473b-5b4698ff58-nhcs4" Sep 12 02:07:31.869837 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:31.869611 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-60c2473b-5b4698ff58-nhcs4" event={"ID":"c4a62cf7-0de1-46b2-9242-0f0d66fca18d","Type":"ContainerDied","Data":"33d3b1b4e4410f5a7d63eb9e56aac3fbbd1a4317b2f9f8c491355b92e0a23e62"} Sep 12 02:07:31.869837 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:31.869677 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-60c2473b-5b4698ff58-nhcs4" event={"ID":"c4a62cf7-0de1-46b2-9242-0f0d66fca18d","Type":"ContainerDied","Data":"ff76f770d317992039c3d698bda5591e05e11d5edd480ad98e08240db018117c"} Sep 12 02:07:31.869837 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:31.869711 2579 scope.go:117] "RemoveContainer" containerID="33d3b1b4e4410f5a7d63eb9e56aac3fbbd1a4317b2f9f8c491355b92e0a23e62" Sep 12 02:07:31.871260 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:31.871237 2579 generic.go:356] "Generic (PLEG): container finished" podID="4adfbf96-ff58-48aa-9604-b6d4463655e2" containerID="b03d221b6267ec7591513bdab12ce287ba8028615066bedf33337309a55fb3c3" exitCode=0 Sep 12 02:07:31.871387 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:31.871279 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-60c2473b-7cfb747df6-flgvv" Sep 12 02:07:31.871387 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:31.871357 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-60c2473b-7cfb747df6-flgvv" event={"ID":"4adfbf96-ff58-48aa-9604-b6d4463655e2","Type":"ContainerDied","Data":"b03d221b6267ec7591513bdab12ce287ba8028615066bedf33337309a55fb3c3"} Sep 12 02:07:31.871515 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:31.871401 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-60c2473b-7cfb747df6-flgvv" event={"ID":"4adfbf96-ff58-48aa-9604-b6d4463655e2","Type":"ContainerDied","Data":"10c51f37027cd22a3add4999978641568a21c445b649f5d9158d9b881358ddb6"} Sep 12 02:07:31.879826 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:31.879809 2579 scope.go:117] "RemoveContainer" containerID="33d3b1b4e4410f5a7d63eb9e56aac3fbbd1a4317b2f9f8c491355b92e0a23e62" Sep 12 02:07:31.880121 ip-10-0-133-79 kubenswrapper[2579]: E0912 02:07:31.880100 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33d3b1b4e4410f5a7d63eb9e56aac3fbbd1a4317b2f9f8c491355b92e0a23e62\": container with ID starting with 33d3b1b4e4410f5a7d63eb9e56aac3fbbd1a4317b2f9f8c491355b92e0a23e62 not found: ID does not exist" containerID="33d3b1b4e4410f5a7d63eb9e56aac3fbbd1a4317b2f9f8c491355b92e0a23e62" Sep 12 02:07:31.880182 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:31.880129 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33d3b1b4e4410f5a7d63eb9e56aac3fbbd1a4317b2f9f8c491355b92e0a23e62"} err="failed to get container status \"33d3b1b4e4410f5a7d63eb9e56aac3fbbd1a4317b2f9f8c491355b92e0a23e62\": rpc error: code = NotFound desc = could not find container \"33d3b1b4e4410f5a7d63eb9e56aac3fbbd1a4317b2f9f8c491355b92e0a23e62\": container with ID starting with 33d3b1b4e4410f5a7d63eb9e56aac3fbbd1a4317b2f9f8c491355b92e0a23e62 not found: ID does not exist" Sep 12 02:07:31.880182 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:31.880146 2579 scope.go:117] "RemoveContainer" containerID="b03d221b6267ec7591513bdab12ce287ba8028615066bedf33337309a55fb3c3" Sep 12 02:07:31.888948 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:31.888921 2579 scope.go:117] "RemoveContainer" containerID="b03d221b6267ec7591513bdab12ce287ba8028615066bedf33337309a55fb3c3" Sep 12 02:07:31.889235 ip-10-0-133-79 kubenswrapper[2579]: E0912 02:07:31.889216 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b03d221b6267ec7591513bdab12ce287ba8028615066bedf33337309a55fb3c3\": container with ID starting with b03d221b6267ec7591513bdab12ce287ba8028615066bedf33337309a55fb3c3 not found: ID does not exist" containerID="b03d221b6267ec7591513bdab12ce287ba8028615066bedf33337309a55fb3c3" Sep 12 02:07:31.889294 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:31.889241 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b03d221b6267ec7591513bdab12ce287ba8028615066bedf33337309a55fb3c3"} err="failed to get container status \"b03d221b6267ec7591513bdab12ce287ba8028615066bedf33337309a55fb3c3\": rpc error: code = NotFound desc = could not find container \"b03d221b6267ec7591513bdab12ce287ba8028615066bedf33337309a55fb3c3\": container with ID starting with b03d221b6267ec7591513bdab12ce287ba8028615066bedf33337309a55fb3c3 not found: ID does not exist" Sep 12 02:07:31.890511 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:31.890422 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-60c2473b-7cfb747df6-flgvv"] Sep 12 02:07:31.892830 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:31.892808 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-60c2473b-7cfb747df6-flgvv"] Sep 12 02:07:31.903594 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:31.903570 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-60c2473b-5b4698ff58-nhcs4"] Sep 12 02:07:31.909320 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:31.909298 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-60c2473b-5b4698ff58-nhcs4"] Sep 12 02:07:32.284394 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:32.284356 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:07:32.919118 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:32.919078 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:07:33.669464 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:33.669418 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:07:33.728316 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:33.728281 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4adfbf96-ff58-48aa-9604-b6d4463655e2" path="/var/lib/kubelet/pods/4adfbf96-ff58-48aa-9604-b6d4463655e2/volumes" Sep 12 02:07:33.728862 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:33.728843 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4a62cf7-0de1-46b2-9242-0f0d66fca18d" path="/var/lib/kubelet/pods/c4a62cf7-0de1-46b2-9242-0f0d66fca18d/volumes" Sep 12 02:07:34.330919 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:34.330882 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:07:34.640585 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:34.640543 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:07:35.047639 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:35.047551 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:07:35.772588 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:35.772551 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:07:39.717158 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:39.717113 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:07:44.430950 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:44.430917 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-98f68e/distinct-2-w4-98f68e-kserve-c5758bc66-d9c6h"] Sep 12 02:07:44.431383 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:44.431370 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4a62cf7-0de1-46b2-9242-0f0d66fca18d" containerName="payload-pre-processing" Sep 12 02:07:44.431428 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:44.431384 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4a62cf7-0de1-46b2-9242-0f0d66fca18d" containerName="payload-pre-processing" Sep 12 02:07:44.431428 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:44.431403 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4adfbf96-ff58-48aa-9604-b6d4463655e2" containerName="maas-api" Sep 12 02:07:44.431428 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:44.431408 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4adfbf96-ff58-48aa-9604-b6d4463655e2" containerName="maas-api" Sep 12 02:07:44.431549 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:44.431491 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4a62cf7-0de1-46b2-9242-0f0d66fca18d" containerName="payload-pre-processing" Sep 12 02:07:44.431549 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:44.431505 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4adfbf96-ff58-48aa-9604-b6d4463655e2" containerName="maas-api" Sep 12 02:07:44.436229 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:44.436208 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-98f68e/distinct-2-w4-98f68e-kserve-c5758bc66-d9c6h" Sep 12 02:07:44.438696 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:44.438676 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-98f68e\"/\"kube-root-ca.crt\"" Sep 12 02:07:44.439604 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:44.439582 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-98f68e\"/\"distinct-2-w4-98f68e-kserve-self-signed-certs\"" Sep 12 02:07:44.439604 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:44.439584 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-98f68e\"/\"default-dockercfg-494kz\"" Sep 12 02:07:44.439770 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:44.439627 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-98f68e\"/\"openshift-service-ca.crt\"" Sep 12 02:07:44.445862 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:44.445834 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-98f68e/distinct-2-w4-98f68e-kserve-c5758bc66-d9c6h"] Sep 12 02:07:44.468664 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:44.468620 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:07:44.531769 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:44.531725 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5-home\") pod \"distinct-2-w4-98f68e-kserve-c5758bc66-d9c6h\" (UID: \"cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5\") " pod="e2e-models-e2e-worker-w4-98f68e/distinct-2-w4-98f68e-kserve-c5758bc66-d9c6h" Sep 12 02:07:44.531769 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:44.531769 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5-model-cache\") pod \"distinct-2-w4-98f68e-kserve-c5758bc66-d9c6h\" (UID: \"cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5\") " pod="e2e-models-e2e-worker-w4-98f68e/distinct-2-w4-98f68e-kserve-c5758bc66-d9c6h" Sep 12 02:07:44.531988 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:44.531794 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5-tls-certs\") pod \"distinct-2-w4-98f68e-kserve-c5758bc66-d9c6h\" (UID: \"cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5\") " pod="e2e-models-e2e-worker-w4-98f68e/distinct-2-w4-98f68e-kserve-c5758bc66-d9c6h" Sep 12 02:07:44.531988 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:44.531847 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5-dshm\") pod \"distinct-2-w4-98f68e-kserve-c5758bc66-d9c6h\" (UID: \"cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5\") " pod="e2e-models-e2e-worker-w4-98f68e/distinct-2-w4-98f68e-kserve-c5758bc66-d9c6h" Sep 12 02:07:44.531988 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:44.531901 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs92k\" (UniqueName: \"kubernetes.io/projected/cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5-kube-api-access-rs92k\") pod \"distinct-2-w4-98f68e-kserve-c5758bc66-d9c6h\" (UID: \"cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5\") " pod="e2e-models-e2e-worker-w4-98f68e/distinct-2-w4-98f68e-kserve-c5758bc66-d9c6h" Sep 12 02:07:44.531988 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:44.531941 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5-tmp-dir\") pod \"distinct-2-w4-98f68e-kserve-c5758bc66-d9c6h\" (UID: \"cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5\") " pod="e2e-models-e2e-worker-w4-98f68e/distinct-2-w4-98f68e-kserve-c5758bc66-d9c6h" Sep 12 02:07:44.633083 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:44.633044 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5-home\") pod \"distinct-2-w4-98f68e-kserve-c5758bc66-d9c6h\" (UID: \"cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5\") " pod="e2e-models-e2e-worker-w4-98f68e/distinct-2-w4-98f68e-kserve-c5758bc66-d9c6h" Sep 12 02:07:44.633083 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:44.633082 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5-model-cache\") pod \"distinct-2-w4-98f68e-kserve-c5758bc66-d9c6h\" (UID: \"cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5\") " pod="e2e-models-e2e-worker-w4-98f68e/distinct-2-w4-98f68e-kserve-c5758bc66-d9c6h" Sep 12 02:07:44.633301 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:44.633121 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5-tls-certs\") pod \"distinct-2-w4-98f68e-kserve-c5758bc66-d9c6h\" (UID: \"cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5\") " pod="e2e-models-e2e-worker-w4-98f68e/distinct-2-w4-98f68e-kserve-c5758bc66-d9c6h" Sep 12 02:07:44.633301 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:44.633173 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5-dshm\") pod \"distinct-2-w4-98f68e-kserve-c5758bc66-d9c6h\" (UID: \"cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5\") " pod="e2e-models-e2e-worker-w4-98f68e/distinct-2-w4-98f68e-kserve-c5758bc66-d9c6h" Sep 12 02:07:44.633301 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:44.633228 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs92k\" (UniqueName: \"kubernetes.io/projected/cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5-kube-api-access-rs92k\") pod \"distinct-2-w4-98f68e-kserve-c5758bc66-d9c6h\" (UID: \"cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5\") " pod="e2e-models-e2e-worker-w4-98f68e/distinct-2-w4-98f68e-kserve-c5758bc66-d9c6h" Sep 12 02:07:44.633301 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:44.633270 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5-tmp-dir\") pod \"distinct-2-w4-98f68e-kserve-c5758bc66-d9c6h\" (UID: \"cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5\") " pod="e2e-models-e2e-worker-w4-98f68e/distinct-2-w4-98f68e-kserve-c5758bc66-d9c6h" Sep 12 02:07:44.633525 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:44.633417 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5-home\") pod \"distinct-2-w4-98f68e-kserve-c5758bc66-d9c6h\" (UID: \"cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5\") " pod="e2e-models-e2e-worker-w4-98f68e/distinct-2-w4-98f68e-kserve-c5758bc66-d9c6h" Sep 12 02:07:44.633659 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:44.633641 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5-tmp-dir\") pod \"distinct-2-w4-98f68e-kserve-c5758bc66-d9c6h\" (UID: \"cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5\") " pod="e2e-models-e2e-worker-w4-98f68e/distinct-2-w4-98f68e-kserve-c5758bc66-d9c6h" Sep 12 02:07:44.633723 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:44.633678 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5-model-cache\") pod \"distinct-2-w4-98f68e-kserve-c5758bc66-d9c6h\" (UID: \"cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5\") " pod="e2e-models-e2e-worker-w4-98f68e/distinct-2-w4-98f68e-kserve-c5758bc66-d9c6h" Sep 12 02:07:44.635734 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:44.635709 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5-dshm\") pod \"distinct-2-w4-98f68e-kserve-c5758bc66-d9c6h\" (UID: \"cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5\") " pod="e2e-models-e2e-worker-w4-98f68e/distinct-2-w4-98f68e-kserve-c5758bc66-d9c6h" Sep 12 02:07:44.635908 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:44.635891 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5-tls-certs\") pod \"distinct-2-w4-98f68e-kserve-c5758bc66-d9c6h\" (UID: \"cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5\") " pod="e2e-models-e2e-worker-w4-98f68e/distinct-2-w4-98f68e-kserve-c5758bc66-d9c6h" Sep 12 02:07:44.641495 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:44.641471 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs92k\" (UniqueName: \"kubernetes.io/projected/cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5-kube-api-access-rs92k\") pod \"distinct-2-w4-98f68e-kserve-c5758bc66-d9c6h\" (UID: \"cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5\") " pod="e2e-models-e2e-worker-w4-98f68e/distinct-2-w4-98f68e-kserve-c5758bc66-d9c6h" Sep 12 02:07:44.747650 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:44.747555 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-98f68e/distinct-2-w4-98f68e-kserve-c5758bc66-d9c6h" Sep 12 02:07:45.022831 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:45.022803 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-98f68e/distinct-2-w4-98f68e-kserve-c5758bc66-d9c6h"] Sep 12 02:07:45.024545 ip-10-0-133-79 kubenswrapper[2579]: W0912 02:07:45.024510 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc84d075_65c5_4dcf_9f6f_6cd0737ef9d5.slice/crio-ddbac34853a815eccb65e66f9f428dd9d85513b2e9bdf2e0a124d9d4ff0153c2 WatchSource:0}: Error finding container ddbac34853a815eccb65e66f9f428dd9d85513b2e9bdf2e0a124d9d4ff0153c2: Status 404 returned error can't find the container with id ddbac34853a815eccb65e66f9f428dd9d85513b2e9bdf2e0a124d9d4ff0153c2 Sep 12 02:07:45.954111 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:45.954069 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-98f68e/distinct-2-w4-98f68e-kserve-c5758bc66-d9c6h" event={"ID":"cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5","Type":"ContainerStarted","Data":"686395bfb451da8345b0f20de213d06db6f065515b7346eb03a99e5d99c24647"} Sep 12 02:07:45.954553 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:45.954117 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-98f68e/distinct-2-w4-98f68e-kserve-c5758bc66-d9c6h" event={"ID":"cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5","Type":"ContainerStarted","Data":"ddbac34853a815eccb65e66f9f428dd9d85513b2e9bdf2e0a124d9d4ff0153c2"} Sep 12 02:07:45.971977 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:45.971923 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-98f68e/distinct-2-w4-98f68e-kserve-c5758bc66-d9c6h" podStartSLOduration=1.971904705 podStartE2EDuration="1.971904705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:07:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:07:45.969992916 +0000 UTC m=+1368.761986712" watchObservedRunningTime="2026-09-12 02:07:45.971904705 +0000 UTC m=+1368.763898500" Sep 12 02:07:54.747912 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:54.747880 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-98f68e/distinct-2-w4-98f68e-kserve-c5758bc66-d9c6h" Sep 12 02:07:54.748366 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:54.747933 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-98f68e/distinct-2-w4-98f68e-kserve-c5758bc66-d9c6h" Sep 12 02:07:54.760985 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:54.760959 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-98f68e/distinct-2-w4-98f68e-kserve-c5758bc66-d9c6h" Sep 12 02:07:55.015857 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:07:55.015782 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-98f68e/distinct-2-w4-98f68e-kserve-c5758bc66-d9c6h" Sep 12 02:08:02.426872 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:08:02.426840 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-fbb6d9/auth-test-model-fbb6d9-kserve-577d59df55-pk7g9"] Sep 12 02:08:02.432025 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:08:02.432006 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-fbb6d9/auth-test-model-fbb6d9-kserve-577d59df55-pk7g9" Sep 12 02:08:02.434810 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:08:02.434785 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-fbb6d9\"/\"openshift-service-ca.crt\"" Sep 12 02:08:02.435717 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:08:02.435699 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-fbb6d9\"/\"kube-root-ca.crt\"" Sep 12 02:08:02.435826 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:08:02.435720 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-fbb6d9\"/\"auth-test-model-fbb6d9-kserve-self-signed-certs\"" Sep 12 02:08:02.435826 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:08:02.435706 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-fbb6d9\"/\"default-dockercfg-mjwfn\"" Sep 12 02:08:02.441723 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:08:02.441698 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-fbb6d9/auth-test-model-fbb6d9-kserve-577d59df55-pk7g9"] Sep 12 02:08:02.454890 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:08:02.454854 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:08:02.515022 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:08:02.514986 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fe29199-dd9d-4326-8a67-d1ca0e11544f-home\") pod \"auth-test-model-fbb6d9-kserve-577d59df55-pk7g9\" (UID: \"8fe29199-dd9d-4326-8a67-d1ca0e11544f\") " pod="ai-tenant-e2e-shared-b-w3-fbb6d9/auth-test-model-fbb6d9-kserve-577d59df55-pk7g9" Sep 12 02:08:02.515022 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:08:02.515026 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fe29199-dd9d-4326-8a67-d1ca0e11544f-tls-certs\") pod \"auth-test-model-fbb6d9-kserve-577d59df55-pk7g9\" (UID: \"8fe29199-dd9d-4326-8a67-d1ca0e11544f\") " pod="ai-tenant-e2e-shared-b-w3-fbb6d9/auth-test-model-fbb6d9-kserve-577d59df55-pk7g9" Sep 12 02:08:02.515225 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:08:02.515118 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fe29199-dd9d-4326-8a67-d1ca0e11544f-tmp-dir\") pod \"auth-test-model-fbb6d9-kserve-577d59df55-pk7g9\" (UID: \"8fe29199-dd9d-4326-8a67-d1ca0e11544f\") " pod="ai-tenant-e2e-shared-b-w3-fbb6d9/auth-test-model-fbb6d9-kserve-577d59df55-pk7g9" Sep 12 02:08:02.515225 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:08:02.515143 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fe29199-dd9d-4326-8a67-d1ca0e11544f-dshm\") pod \"auth-test-model-fbb6d9-kserve-577d59df55-pk7g9\" (UID: \"8fe29199-dd9d-4326-8a67-d1ca0e11544f\") " pod="ai-tenant-e2e-shared-b-w3-fbb6d9/auth-test-model-fbb6d9-kserve-577d59df55-pk7g9" Sep 12 02:08:02.515298 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:08:02.515230 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ffqz\" (UniqueName: \"kubernetes.io/projected/8fe29199-dd9d-4326-8a67-d1ca0e11544f-kube-api-access-8ffqz\") pod \"auth-test-model-fbb6d9-kserve-577d59df55-pk7g9\" (UID: \"8fe29199-dd9d-4326-8a67-d1ca0e11544f\") " pod="ai-tenant-e2e-shared-b-w3-fbb6d9/auth-test-model-fbb6d9-kserve-577d59df55-pk7g9" Sep 12 02:08:02.515333 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:08:02.515317 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fe29199-dd9d-4326-8a67-d1ca0e11544f-model-cache\") pod \"auth-test-model-fbb6d9-kserve-577d59df55-pk7g9\" (UID: \"8fe29199-dd9d-4326-8a67-d1ca0e11544f\") " pod="ai-tenant-e2e-shared-b-w3-fbb6d9/auth-test-model-fbb6d9-kserve-577d59df55-pk7g9" Sep 12 02:08:02.616765 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:08:02.616732 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fe29199-dd9d-4326-8a67-d1ca0e11544f-home\") pod \"auth-test-model-fbb6d9-kserve-577d59df55-pk7g9\" (UID: \"8fe29199-dd9d-4326-8a67-d1ca0e11544f\") " pod="ai-tenant-e2e-shared-b-w3-fbb6d9/auth-test-model-fbb6d9-kserve-577d59df55-pk7g9" Sep 12 02:08:02.616765 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:08:02.616771 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fe29199-dd9d-4326-8a67-d1ca0e11544f-tls-certs\") pod \"auth-test-model-fbb6d9-kserve-577d59df55-pk7g9\" (UID: \"8fe29199-dd9d-4326-8a67-d1ca0e11544f\") " pod="ai-tenant-e2e-shared-b-w3-fbb6d9/auth-test-model-fbb6d9-kserve-577d59df55-pk7g9" Sep 12 02:08:02.617002 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:08:02.616827 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fe29199-dd9d-4326-8a67-d1ca0e11544f-tmp-dir\") pod \"auth-test-model-fbb6d9-kserve-577d59df55-pk7g9\" (UID: \"8fe29199-dd9d-4326-8a67-d1ca0e11544f\") " pod="ai-tenant-e2e-shared-b-w3-fbb6d9/auth-test-model-fbb6d9-kserve-577d59df55-pk7g9" Sep 12 02:08:02.617002 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:08:02.616848 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fe29199-dd9d-4326-8a67-d1ca0e11544f-dshm\") pod \"auth-test-model-fbb6d9-kserve-577d59df55-pk7g9\" (UID: \"8fe29199-dd9d-4326-8a67-d1ca0e11544f\") " pod="ai-tenant-e2e-shared-b-w3-fbb6d9/auth-test-model-fbb6d9-kserve-577d59df55-pk7g9" Sep 12 02:08:02.617002 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:08:02.616888 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ffqz\" (UniqueName: \"kubernetes.io/projected/8fe29199-dd9d-4326-8a67-d1ca0e11544f-kube-api-access-8ffqz\") pod \"auth-test-model-fbb6d9-kserve-577d59df55-pk7g9\" (UID: \"8fe29199-dd9d-4326-8a67-d1ca0e11544f\") " pod="ai-tenant-e2e-shared-b-w3-fbb6d9/auth-test-model-fbb6d9-kserve-577d59df55-pk7g9" Sep 12 02:08:02.617002 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:08:02.616969 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fe29199-dd9d-4326-8a67-d1ca0e11544f-model-cache\") pod \"auth-test-model-fbb6d9-kserve-577d59df55-pk7g9\" (UID: \"8fe29199-dd9d-4326-8a67-d1ca0e11544f\") " pod="ai-tenant-e2e-shared-b-w3-fbb6d9/auth-test-model-fbb6d9-kserve-577d59df55-pk7g9" Sep 12 02:08:02.617204 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:08:02.617184 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fe29199-dd9d-4326-8a67-d1ca0e11544f-home\") pod \"auth-test-model-fbb6d9-kserve-577d59df55-pk7g9\" (UID: \"8fe29199-dd9d-4326-8a67-d1ca0e11544f\") " pod="ai-tenant-e2e-shared-b-w3-fbb6d9/auth-test-model-fbb6d9-kserve-577d59df55-pk7g9" Sep 12 02:08:02.617204 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:08:02.617194 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fe29199-dd9d-4326-8a67-d1ca0e11544f-tmp-dir\") pod \"auth-test-model-fbb6d9-kserve-577d59df55-pk7g9\" (UID: \"8fe29199-dd9d-4326-8a67-d1ca0e11544f\") " pod="ai-tenant-e2e-shared-b-w3-fbb6d9/auth-test-model-fbb6d9-kserve-577d59df55-pk7g9" Sep 12 02:08:02.617381 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:08:02.617353 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fe29199-dd9d-4326-8a67-d1ca0e11544f-model-cache\") pod \"auth-test-model-fbb6d9-kserve-577d59df55-pk7g9\" (UID: \"8fe29199-dd9d-4326-8a67-d1ca0e11544f\") " pod="ai-tenant-e2e-shared-b-w3-fbb6d9/auth-test-model-fbb6d9-kserve-577d59df55-pk7g9" Sep 12 02:08:02.619195 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:08:02.619176 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fe29199-dd9d-4326-8a67-d1ca0e11544f-dshm\") pod \"auth-test-model-fbb6d9-kserve-577d59df55-pk7g9\" (UID: \"8fe29199-dd9d-4326-8a67-d1ca0e11544f\") " pod="ai-tenant-e2e-shared-b-w3-fbb6d9/auth-test-model-fbb6d9-kserve-577d59df55-pk7g9" Sep 12 02:08:02.619611 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:08:02.619591 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fe29199-dd9d-4326-8a67-d1ca0e11544f-tls-certs\") pod \"auth-test-model-fbb6d9-kserve-577d59df55-pk7g9\" (UID: \"8fe29199-dd9d-4326-8a67-d1ca0e11544f\") " pod="ai-tenant-e2e-shared-b-w3-fbb6d9/auth-test-model-fbb6d9-kserve-577d59df55-pk7g9" Sep 12 02:08:02.625835 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:08:02.625810 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ffqz\" (UniqueName: \"kubernetes.io/projected/8fe29199-dd9d-4326-8a67-d1ca0e11544f-kube-api-access-8ffqz\") pod \"auth-test-model-fbb6d9-kserve-577d59df55-pk7g9\" (UID: \"8fe29199-dd9d-4326-8a67-d1ca0e11544f\") " pod="ai-tenant-e2e-shared-b-w3-fbb6d9/auth-test-model-fbb6d9-kserve-577d59df55-pk7g9" Sep 12 02:08:02.743197 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:08:02.743110 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-fbb6d9/auth-test-model-fbb6d9-kserve-577d59df55-pk7g9" Sep 12 02:08:03.017493 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:08:03.017401 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-fbb6d9/auth-test-model-fbb6d9-kserve-577d59df55-pk7g9"] Sep 12 02:08:03.020508 ip-10-0-133-79 kubenswrapper[2579]: W0912 02:08:03.020481 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8fe29199_dd9d_4326_8a67_d1ca0e11544f.slice/crio-82933601679beb1eb76f0c3c3c9ca49349f523aa9621492627643de14c3d1668 WatchSource:0}: Error finding container 82933601679beb1eb76f0c3c3c9ca49349f523aa9621492627643de14c3d1668: Status 404 returned error can't find the container with id 82933601679beb1eb76f0c3c3c9ca49349f523aa9621492627643de14c3d1668 Sep 12 02:08:03.047927 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:08:03.047898 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-fbb6d9/auth-test-model-fbb6d9-kserve-577d59df55-pk7g9" event={"ID":"8fe29199-dd9d-4326-8a67-d1ca0e11544f","Type":"ContainerStarted","Data":"82933601679beb1eb76f0c3c3c9ca49349f523aa9621492627643de14c3d1668"} Sep 12 02:08:03.129471 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:08:03.129418 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:08:04.055829 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:08:04.055797 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-fbb6d9/auth-test-model-fbb6d9-kserve-577d59df55-pk7g9" event={"ID":"8fe29199-dd9d-4326-8a67-d1ca0e11544f","Type":"ContainerStarted","Data":"d98c0f71c40a698b9c7576d771f0efb0374349efe4cb4f2f863b43f700bdcc2e"} Sep 12 02:08:04.075296 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:08:04.075232 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-fbb6d9/auth-test-model-fbb6d9-kserve-577d59df55-pk7g9" podStartSLOduration=2.075214721 podStartE2EDuration="2.075214721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:08:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:08:04.07383744 +0000 UTC m=+1386.865831235" watchObservedRunningTime="2026-09-12 02:08:04.075214721 +0000 UTC m=+1386.867208515" Sep 12 02:08:10.179437 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:08:10.179400 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:08:10.237418 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:08:10.237384 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:08:12.743620 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:08:12.743587 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-fbb6d9/auth-test-model-fbb6d9-kserve-577d59df55-pk7g9" Sep 12 02:08:12.744112 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:08:12.743639 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-fbb6d9/auth-test-model-fbb6d9-kserve-577d59df55-pk7g9" Sep 12 02:08:12.756582 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:08:12.756553 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-fbb6d9/auth-test-model-fbb6d9-kserve-577d59df55-pk7g9" Sep 12 02:08:13.117711 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:08:13.117630 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-fbb6d9/auth-test-model-fbb6d9-kserve-577d59df55-pk7g9" Sep 12 02:08:16.463669 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:08:16.463630 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:08:16.901634 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:08:16.901587 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:08:22.368601 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:08:22.368566 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:08:32.790383 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:08:32.790304 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:08:36.537564 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:08:36.537525 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:08:36.686089 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:08:36.686049 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:08:36.756951 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:08:36.756916 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:08:36.955843 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:08:36.955814 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:08:37.950583 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:08:37.950548 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:08:38.257497 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:08:38.257387 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:08:38.353728 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:08:38.353694 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:08:38.449609 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:08:38.449572 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:08:39.086636 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:08:39.086603 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:08:39.182726 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:08:39.182681 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:08:39.283183 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:08:39.283154 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:08:40.627875 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:08:40.627841 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:08:46.366526 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:08:46.366489 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:08:46.468474 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:08:46.465598 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:08:47.119442 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:08:47.119406 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:08:48.452225 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:08:48.452193 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:08:52.731868 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:08:52.731817 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:08:52.793072 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:08:52.793025 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:08:52.864160 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:08:52.864111 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:08:52.964367 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:08:52.964329 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:09:03.435912 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:09:03.435876 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:09:03.676047 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:09:03.676005 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:09:05.089443 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:09:05.089403 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:09:05.150669 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:09:05.150634 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:09:11.530764 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:09:11.530723 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:09:13.277180 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:09:13.277146 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:09:25.442954 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:09:25.442915 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:09:30.695627 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:09:30.695588 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:09:30.794953 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:09:30.794921 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:09:31.099610 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:09:31.099503 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:09:31.377088 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:09:31.377052 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:09:31.705020 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:09:31.704916 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:09:33.657312 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:09:33.657274 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:09:34.451770 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:09:34.451709 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:09:34.545948 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:09:34.545907 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:09:34.642863 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:09:34.642828 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:09:35.161223 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:09:35.161189 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:09:35.951619 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:09:35.951581 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:09:36.596997 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:09:36.596967 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:09:37.250109 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:09:37.250077 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:09:38.048794 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:09:38.048755 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:09:38.790863 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:09:38.790832 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:09:38.852636 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:09:38.852601 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:09:38.987663 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:09:38.987623 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:09:39.237391 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:09:39.237361 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:09:39.338152 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:09:39.338113 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:09:39.340003 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:09:39.339980 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-caecee-659b9c48cd-tvr4c"] Sep 12 02:09:39.344066 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:09:39.344045 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-caecee-659b9c48cd-tvr4c" Sep 12 02:09:39.346682 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:09:39.346661 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-caecee-dockercfg-6lnws\"" Sep 12 02:09:39.346794 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:09:39.346668 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-caecee\"" Sep 12 02:09:39.360653 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:09:39.360620 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-caecee-659b9c48cd-tvr4c"] Sep 12 02:09:39.425894 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:09:39.425859 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/132fc085-122b-48ab-b07e-dd81c6c1c516-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-caecee-659b9c48cd-tvr4c\" (UID: \"132fc085-122b-48ab-b07e-dd81c6c1c516\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-caecee-659b9c48cd-tvr4c" Sep 12 02:09:39.426082 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:09:39.425945 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkg9d\" (UniqueName: \"kubernetes.io/projected/132fc085-122b-48ab-b07e-dd81c6c1c516-kube-api-access-bkg9d\") pod \"payload-pre-processing-e2e-api-a-caecee-659b9c48cd-tvr4c\" (UID: \"132fc085-122b-48ab-b07e-dd81c6c1c516\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-caecee-659b9c48cd-tvr4c" Sep 12 02:09:39.527245 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:09:39.527152 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/132fc085-122b-48ab-b07e-dd81c6c1c516-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-caecee-659b9c48cd-tvr4c\" (UID: \"132fc085-122b-48ab-b07e-dd81c6c1c516\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-caecee-659b9c48cd-tvr4c" Sep 12 02:09:39.527245 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:09:39.527225 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkg9d\" (UniqueName: \"kubernetes.io/projected/132fc085-122b-48ab-b07e-dd81c6c1c516-kube-api-access-bkg9d\") pod \"payload-pre-processing-e2e-api-a-caecee-659b9c48cd-tvr4c\" (UID: \"132fc085-122b-48ab-b07e-dd81c6c1c516\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-caecee-659b9c48cd-tvr4c" Sep 12 02:09:39.527912 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:09:39.527891 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/132fc085-122b-48ab-b07e-dd81c6c1c516-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-caecee-659b9c48cd-tvr4c\" (UID: \"132fc085-122b-48ab-b07e-dd81c6c1c516\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-caecee-659b9c48cd-tvr4c" Sep 12 02:09:39.535001 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:09:39.534974 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkg9d\" (UniqueName: \"kubernetes.io/projected/132fc085-122b-48ab-b07e-dd81c6c1c516-kube-api-access-bkg9d\") pod \"payload-pre-processing-e2e-api-a-caecee-659b9c48cd-tvr4c\" (UID: \"132fc085-122b-48ab-b07e-dd81c6c1c516\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-caecee-659b9c48cd-tvr4c" Sep 12 02:09:39.655815 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:09:39.655772 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-caecee-659b9c48cd-tvr4c" Sep 12 02:09:39.929187 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:09:39.929159 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-caecee-659b9c48cd-tvr4c"] Sep 12 02:09:39.931135 ip-10-0-133-79 kubenswrapper[2579]: W0912 02:09:39.931108 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod132fc085_122b_48ab_b07e_dd81c6c1c516.slice/crio-7edcbfe5791413426d1c0133337f7b64daf9f441677db60a85d9e2e7bef976fd WatchSource:0}: Error finding container 7edcbfe5791413426d1c0133337f7b64daf9f441677db60a85d9e2e7bef976fd: Status 404 returned error can't find the container with id 7edcbfe5791413426d1c0133337f7b64daf9f441677db60a85d9e2e7bef976fd Sep 12 02:09:40.598081 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:09:40.598040 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-caecee-659b9c48cd-tvr4c" event={"ID":"132fc085-122b-48ab-b07e-dd81c6c1c516","Type":"ContainerStarted","Data":"a2d5561aaccc545041e1846c7225d9dd098069d795884594755f0c0e135e9790"} Sep 12 02:09:40.598533 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:09:40.598108 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-caecee-659b9c48cd-tvr4c" event={"ID":"132fc085-122b-48ab-b07e-dd81c6c1c516","Type":"ContainerStarted","Data":"7edcbfe5791413426d1c0133337f7b64daf9f441677db60a85d9e2e7bef976fd"} Sep 12 02:09:40.598533 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:09:40.598157 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-caecee-659b9c48cd-tvr4c" Sep 12 02:09:40.617165 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:09:40.617111 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-caecee-659b9c48cd-tvr4c" podStartSLOduration=1.617092993 podStartE2EDuration="1.617092993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:09:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:09:40.616031063 +0000 UTC m=+1483.408024871" watchObservedRunningTime="2026-09-12 02:09:40.617092993 +0000 UTC m=+1483.409086830" Sep 12 02:09:45.577954 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:09:45.577917 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:09:49.023880 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:09:49.023814 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:09:50.533503 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:09:50.533467 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:09:51.606638 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:09:51.606607 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-caecee-659b9c48cd-tvr4c" Sep 12 02:09:59.548656 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:09:59.548608 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:09:59.650567 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:09:59.650525 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:00.527282 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:00.527242 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:01.193890 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:01.193855 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:01.295665 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:01.295626 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:02.171821 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:02.171782 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:02.736039 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:02.736004 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:03.564897 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:03.564859 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:04.298848 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:04.298813 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:04.496391 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:04.496357 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:04.603284 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:04.603190 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f660a6/distinct-2-w0-f660a6-kserve-5d76f79fd5-ksmq5"] Sep 12 02:10:04.639258 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:04.639223 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f660a6/simulator-w0-f660a6-kserve-779fbfd878-4wgwt"] Sep 12 02:10:04.698030 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:04.697993 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:04.984098 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:04.984056 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:05.116816 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:05.116779 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:05.328086 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:05.327991 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:05.724036 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:05.723991 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:06.112087 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:06.112001 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:07.002409 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:07.002368 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:07.963157 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:07.962894 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:08.381002 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:08.380965 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:08.479724 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:08.479683 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:08.951696 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:08.951666 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:09.958527 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:09.958494 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:10.711282 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:10.711250 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:11.453984 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:11.453954 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:12.156761 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:12.156727 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:12.850889 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:12.850855 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:13.731047 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:13.731004 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b23bee-774f64cf68-hsjbn"] Sep 12 02:10:13.736533 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:13.736504 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b23bee-774f64cf68-hsjbn"] Sep 12 02:10:13.736717 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:13.736607 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b23bee-774f64cf68-hsjbn" Sep 12 02:10:13.738716 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:13.738692 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-b23bee-serving-cert\"" Sep 12 02:10:13.757216 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:13.757186 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:13.778689 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:13.778651 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsmk6\" (UniqueName: \"kubernetes.io/projected/9c9f8ad4-58c8-45a9-8183-00a5976e1f54-kube-api-access-jsmk6\") pod \"maas-api-e2e-api-b-b23bee-774f64cf68-hsjbn\" (UID: \"9c9f8ad4-58c8-45a9-8183-00a5976e1f54\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b23bee-774f64cf68-hsjbn" Sep 12 02:10:13.778954 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:13.778928 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c9f8ad4-58c8-45a9-8183-00a5976e1f54-metrics-certs\") pod \"maas-api-e2e-api-b-b23bee-774f64cf68-hsjbn\" (UID: \"9c9f8ad4-58c8-45a9-8183-00a5976e1f54\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b23bee-774f64cf68-hsjbn" Sep 12 02:10:13.779365 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:13.779336 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c9f8ad4-58c8-45a9-8183-00a5976e1f54-maas-api-tls\") pod \"maas-api-e2e-api-b-b23bee-774f64cf68-hsjbn\" (UID: \"9c9f8ad4-58c8-45a9-8183-00a5976e1f54\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b23bee-774f64cf68-hsjbn" Sep 12 02:10:13.779607 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:13.779583 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c9f8ad4-58c8-45a9-8183-00a5976e1f54-openshift-service-ca\") pod \"maas-api-e2e-api-b-b23bee-774f64cf68-hsjbn\" (UID: \"9c9f8ad4-58c8-45a9-8183-00a5976e1f54\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b23bee-774f64cf68-hsjbn" Sep 12 02:10:13.857229 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:13.857189 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:13.880839 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:13.880806 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsmk6\" (UniqueName: \"kubernetes.io/projected/9c9f8ad4-58c8-45a9-8183-00a5976e1f54-kube-api-access-jsmk6\") pod \"maas-api-e2e-api-b-b23bee-774f64cf68-hsjbn\" (UID: \"9c9f8ad4-58c8-45a9-8183-00a5976e1f54\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b23bee-774f64cf68-hsjbn" Sep 12 02:10:13.880839 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:13.880851 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c9f8ad4-58c8-45a9-8183-00a5976e1f54-metrics-certs\") pod \"maas-api-e2e-api-b-b23bee-774f64cf68-hsjbn\" (UID: \"9c9f8ad4-58c8-45a9-8183-00a5976e1f54\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b23bee-774f64cf68-hsjbn" Sep 12 02:10:13.881064 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:13.880952 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c9f8ad4-58c8-45a9-8183-00a5976e1f54-maas-api-tls\") pod \"maas-api-e2e-api-b-b23bee-774f64cf68-hsjbn\" (UID: \"9c9f8ad4-58c8-45a9-8183-00a5976e1f54\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b23bee-774f64cf68-hsjbn" Sep 12 02:10:13.881064 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:13.881007 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c9f8ad4-58c8-45a9-8183-00a5976e1f54-openshift-service-ca\") pod \"maas-api-e2e-api-b-b23bee-774f64cf68-hsjbn\" (UID: \"9c9f8ad4-58c8-45a9-8183-00a5976e1f54\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b23bee-774f64cf68-hsjbn" Sep 12 02:10:13.881171 ip-10-0-133-79 kubenswrapper[2579]: E0912 02:10:13.881122 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-b23bee-serving-cert: secret "maas-api-e2e-api-b-b23bee-serving-cert" not found Sep 12 02:10:13.881227 ip-10-0-133-79 kubenswrapper[2579]: E0912 02:10:13.881216 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c9f8ad4-58c8-45a9-8183-00a5976e1f54-maas-api-tls podName:9c9f8ad4-58c8-45a9-8183-00a5976e1f54 nodeName:}" failed. No retries permitted until 2026-09-12 02:10:14.381192036 +0000 UTC m=+1517.173186016 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9c9f8ad4-58c8-45a9-8183-00a5976e1f54-maas-api-tls") pod "maas-api-e2e-api-b-b23bee-774f64cf68-hsjbn" (UID: "9c9f8ad4-58c8-45a9-8183-00a5976e1f54") : secret "maas-api-e2e-api-b-b23bee-serving-cert" not found Sep 12 02:10:13.881754 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:13.881734 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c9f8ad4-58c8-45a9-8183-00a5976e1f54-openshift-service-ca\") pod \"maas-api-e2e-api-b-b23bee-774f64cf68-hsjbn\" (UID: \"9c9f8ad4-58c8-45a9-8183-00a5976e1f54\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b23bee-774f64cf68-hsjbn" Sep 12 02:10:13.883432 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:13.883410 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c9f8ad4-58c8-45a9-8183-00a5976e1f54-metrics-certs\") pod \"maas-api-e2e-api-b-b23bee-774f64cf68-hsjbn\" (UID: \"9c9f8ad4-58c8-45a9-8183-00a5976e1f54\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b23bee-774f64cf68-hsjbn" Sep 12 02:10:13.891628 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:13.891607 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsmk6\" (UniqueName: \"kubernetes.io/projected/9c9f8ad4-58c8-45a9-8183-00a5976e1f54-kube-api-access-jsmk6\") pod \"maas-api-e2e-api-b-b23bee-774f64cf68-hsjbn\" (UID: \"9c9f8ad4-58c8-45a9-8183-00a5976e1f54\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b23bee-774f64cf68-hsjbn" Sep 12 02:10:13.986400 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:13.986292 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:14.236955 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:14.236872 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:14.343366 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:14.343329 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:14.350177 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:14.350151 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b23bee-898989fc4-kz7qg"] Sep 12 02:10:14.354031 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:14.354013 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b23bee-898989fc4-kz7qg" Sep 12 02:10:14.356427 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:14.356408 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-b23bee-dockercfg-7tqst\"" Sep 12 02:10:14.356695 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:14.356680 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-b23bee\"" Sep 12 02:10:14.363028 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:14.363003 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b23bee-898989fc4-kz7qg"] Sep 12 02:10:14.387706 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:14.387674 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c9f8ad4-58c8-45a9-8183-00a5976e1f54-maas-api-tls\") pod \"maas-api-e2e-api-b-b23bee-774f64cf68-hsjbn\" (UID: \"9c9f8ad4-58c8-45a9-8183-00a5976e1f54\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b23bee-774f64cf68-hsjbn" Sep 12 02:10:14.387916 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:14.387767 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvm4w\" (UniqueName: \"kubernetes.io/projected/3f1890ff-65c0-4771-9d6b-6887073fa37f-kube-api-access-mvm4w\") pod \"payload-pre-processing-e2e-api-b-b23bee-898989fc4-kz7qg\" (UID: \"3f1890ff-65c0-4771-9d6b-6887073fa37f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b23bee-898989fc4-kz7qg" Sep 12 02:10:14.387916 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:14.387840 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f1890ff-65c0-4771-9d6b-6887073fa37f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b23bee-898989fc4-kz7qg\" (UID: \"3f1890ff-65c0-4771-9d6b-6887073fa37f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b23bee-898989fc4-kz7qg" Sep 12 02:10:14.390372 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:14.390345 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c9f8ad4-58c8-45a9-8183-00a5976e1f54-maas-api-tls\") pod \"maas-api-e2e-api-b-b23bee-774f64cf68-hsjbn\" (UID: \"9c9f8ad4-58c8-45a9-8183-00a5976e1f54\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b23bee-774f64cf68-hsjbn" Sep 12 02:10:14.489348 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:14.489248 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f1890ff-65c0-4771-9d6b-6887073fa37f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b23bee-898989fc4-kz7qg\" (UID: \"3f1890ff-65c0-4771-9d6b-6887073fa37f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b23bee-898989fc4-kz7qg" Sep 12 02:10:14.489566 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:14.489401 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvm4w\" (UniqueName: \"kubernetes.io/projected/3f1890ff-65c0-4771-9d6b-6887073fa37f-kube-api-access-mvm4w\") pod \"payload-pre-processing-e2e-api-b-b23bee-898989fc4-kz7qg\" (UID: \"3f1890ff-65c0-4771-9d6b-6887073fa37f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b23bee-898989fc4-kz7qg" Sep 12 02:10:14.490121 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:14.490102 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f1890ff-65c0-4771-9d6b-6887073fa37f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b23bee-898989fc4-kz7qg\" (UID: \"3f1890ff-65c0-4771-9d6b-6887073fa37f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b23bee-898989fc4-kz7qg" Sep 12 02:10:14.497625 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:14.497598 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvm4w\" (UniqueName: \"kubernetes.io/projected/3f1890ff-65c0-4771-9d6b-6887073fa37f-kube-api-access-mvm4w\") pod \"payload-pre-processing-e2e-api-b-b23bee-898989fc4-kz7qg\" (UID: \"3f1890ff-65c0-4771-9d6b-6887073fa37f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b23bee-898989fc4-kz7qg" Sep 12 02:10:14.650780 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:14.650737 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b23bee-774f64cf68-hsjbn" Sep 12 02:10:14.666021 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:14.665988 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b23bee-898989fc4-kz7qg" Sep 12 02:10:14.927327 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:14.927296 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b23bee-774f64cf68-hsjbn"] Sep 12 02:10:14.929280 ip-10-0-133-79 kubenswrapper[2579]: W0912 02:10:14.929244 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c9f8ad4_58c8_45a9_8183_00a5976e1f54.slice/crio-13119859bc3c0f3a2cbb6bd8c1d42059a8c8a10ceca6a3fcc95e97525c1b7bb8 WatchSource:0}: Error finding container 13119859bc3c0f3a2cbb6bd8c1d42059a8c8a10ceca6a3fcc95e97525c1b7bb8: Status 404 returned error can't find the container with id 13119859bc3c0f3a2cbb6bd8c1d42059a8c8a10ceca6a3fcc95e97525c1b7bb8 Sep 12 02:10:14.930887 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:14.930864 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 12 02:10:14.952615 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:14.952589 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b23bee-898989fc4-kz7qg"] Sep 12 02:10:14.954602 ip-10-0-133-79 kubenswrapper[2579]: W0912 02:10:14.954574 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f1890ff_65c0_4771_9d6b_6887073fa37f.slice/crio-c2b87cfc0a47fe1a1cb382faa913715b8f96f2799eba28bf27874241e53cd3bc WatchSource:0}: Error finding container c2b87cfc0a47fe1a1cb382faa913715b8f96f2799eba28bf27874241e53cd3bc: Status 404 returned error can't find the container with id c2b87cfc0a47fe1a1cb382faa913715b8f96f2799eba28bf27874241e53cd3bc Sep 12 02:10:15.819277 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:15.819235 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b23bee-774f64cf68-hsjbn" event={"ID":"9c9f8ad4-58c8-45a9-8183-00a5976e1f54","Type":"ContainerStarted","Data":"296d5d296e41ad934b0289c2c127a68dab587b14344d69250312545d086c1f45"} Sep 12 02:10:15.819475 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:15.819406 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b23bee-774f64cf68-hsjbn" Sep 12 02:10:15.819561 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:15.819476 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b23bee-774f64cf68-hsjbn" event={"ID":"9c9f8ad4-58c8-45a9-8183-00a5976e1f54","Type":"ContainerStarted","Data":"13119859bc3c0f3a2cbb6bd8c1d42059a8c8a10ceca6a3fcc95e97525c1b7bb8"} Sep 12 02:10:15.820923 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:15.820901 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b23bee-898989fc4-kz7qg" event={"ID":"3f1890ff-65c0-4771-9d6b-6887073fa37f","Type":"ContainerStarted","Data":"96580acd2335a93fec7984926895ebba7678ad7074f20890982b2c58baad6375"} Sep 12 02:10:15.821054 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:15.820941 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b23bee-898989fc4-kz7qg" event={"ID":"3f1890ff-65c0-4771-9d6b-6887073fa37f","Type":"ContainerStarted","Data":"c2b87cfc0a47fe1a1cb382faa913715b8f96f2799eba28bf27874241e53cd3bc"} Sep 12 02:10:15.821054 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:15.821008 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b23bee-898989fc4-kz7qg" Sep 12 02:10:15.838526 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:15.838475 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b23bee-774f64cf68-hsjbn" podStartSLOduration=2.44526234 podStartE2EDuration="2.838444662s" totalImagesPullingTime="393.182322ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:10:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:10:15.836412897 +0000 UTC m=+1518.628406704" watchObservedRunningTime="2026-09-12 02:10:15.838444662 +0000 UTC m=+1518.630438460" Sep 12 02:10:15.853660 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:15.853610 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b23bee-898989fc4-kz7qg" podStartSLOduration=1.853593278 podStartE2EDuration="1.853593278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:10:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:10:15.851502257 +0000 UTC m=+1518.643496051" watchObservedRunningTime="2026-09-12 02:10:15.853593278 +0000 UTC m=+1518.645587102" Sep 12 02:10:18.455568 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:18.455529 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f660a6-29819640-pfmd7"] Sep 12 02:10:18.456546 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:18.456513 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:18.458336 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:18.458312 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f660a6-29819640-pfmd7"] Sep 12 02:10:18.483168 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:18.483127 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f660a6-7c79ccd4b6-l6sfk"] Sep 12 02:10:18.483431 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:18.483401 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f660a6-7c79ccd4b6-l6sfk" podUID="c273cea1-7278-4513-8432-ccba4c25c999" containerName="payload-pre-processing" containerID="cri-o://491e9a7275f615f2517a7ae00fda6023c4bd0e406fac9d4a0e3e89383cb1cc3c" gracePeriod=30 Sep 12 02:10:18.747776 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:18.747748 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f660a6-7c79ccd4b6-l6sfk" Sep 12 02:10:18.838318 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:18.838285 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2bmqt\" (UniqueName: \"kubernetes.io/projected/c273cea1-7278-4513-8432-ccba4c25c999-kube-api-access-2bmqt\") pod \"c273cea1-7278-4513-8432-ccba4c25c999\" (UID: \"c273cea1-7278-4513-8432-ccba4c25c999\") " Sep 12 02:10:18.838545 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:18.838422 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c273cea1-7278-4513-8432-ccba4c25c999-plugins-config-volume\") pod \"c273cea1-7278-4513-8432-ccba4c25c999\" (UID: \"c273cea1-7278-4513-8432-ccba4c25c999\") " Sep 12 02:10:18.838860 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:18.838819 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c273cea1-7278-4513-8432-ccba4c25c999-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c273cea1-7278-4513-8432-ccba4c25c999" (UID: "c273cea1-7278-4513-8432-ccba4c25c999"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 12 02:10:18.838967 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:18.838927 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c273cea1-7278-4513-8432-ccba4c25c999-plugins-config-volume\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:10:18.840316 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:18.840293 2579 generic.go:356] "Generic (PLEG): container finished" podID="c273cea1-7278-4513-8432-ccba4c25c999" containerID="491e9a7275f615f2517a7ae00fda6023c4bd0e406fac9d4a0e3e89383cb1cc3c" exitCode=0 Sep 12 02:10:18.840401 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:18.840346 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f660a6-7c79ccd4b6-l6sfk" Sep 12 02:10:18.840401 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:18.840383 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f660a6-7c79ccd4b6-l6sfk" event={"ID":"c273cea1-7278-4513-8432-ccba4c25c999","Type":"ContainerDied","Data":"491e9a7275f615f2517a7ae00fda6023c4bd0e406fac9d4a0e3e89383cb1cc3c"} Sep 12 02:10:18.840677 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:18.840491 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f660a6-7c79ccd4b6-l6sfk" event={"ID":"c273cea1-7278-4513-8432-ccba4c25c999","Type":"ContainerDied","Data":"5044056010be35fdc691bba838a5745e9f8730535655e23769738d06a8e27aa4"} Sep 12 02:10:18.840677 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:18.840530 2579 scope.go:117] "RemoveContainer" containerID="491e9a7275f615f2517a7ae00fda6023c4bd0e406fac9d4a0e3e89383cb1cc3c" Sep 12 02:10:18.841180 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:18.841155 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c273cea1-7278-4513-8432-ccba4c25c999-kube-api-access-2bmqt" (OuterVolumeSpecName: "kube-api-access-2bmqt") pod "c273cea1-7278-4513-8432-ccba4c25c999" (UID: "c273cea1-7278-4513-8432-ccba4c25c999"). InnerVolumeSpecName "kube-api-access-2bmqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 12 02:10:18.856530 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:18.856508 2579 scope.go:117] "RemoveContainer" containerID="491e9a7275f615f2517a7ae00fda6023c4bd0e406fac9d4a0e3e89383cb1cc3c" Sep 12 02:10:18.856888 ip-10-0-133-79 kubenswrapper[2579]: E0912 02:10:18.856870 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"491e9a7275f615f2517a7ae00fda6023c4bd0e406fac9d4a0e3e89383cb1cc3c\": container with ID starting with 491e9a7275f615f2517a7ae00fda6023c4bd0e406fac9d4a0e3e89383cb1cc3c not found: ID does not exist" containerID="491e9a7275f615f2517a7ae00fda6023c4bd0e406fac9d4a0e3e89383cb1cc3c" Sep 12 02:10:18.856934 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:18.856898 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"491e9a7275f615f2517a7ae00fda6023c4bd0e406fac9d4a0e3e89383cb1cc3c"} err="failed to get container status \"491e9a7275f615f2517a7ae00fda6023c4bd0e406fac9d4a0e3e89383cb1cc3c\": rpc error: code = NotFound desc = could not find container \"491e9a7275f615f2517a7ae00fda6023c4bd0e406fac9d4a0e3e89383cb1cc3c\": container with ID starting with 491e9a7275f615f2517a7ae00fda6023c4bd0e406fac9d4a0e3e89383cb1cc3c not found: ID does not exist" Sep 12 02:10:18.940278 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:18.940232 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bmqt\" (UniqueName: \"kubernetes.io/projected/c273cea1-7278-4513-8432-ccba4c25c999-kube-api-access-2bmqt\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:10:19.162509 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:19.162473 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f660a6-7c79ccd4b6-l6sfk"] Sep 12 02:10:19.165759 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:19.165727 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f660a6-7c79ccd4b6-l6sfk"] Sep 12 02:10:19.624114 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:19.624057 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f660a6/distinct-2-w0-f660a6-kserve-5d76f79fd5-ksmq5" podUID="6c25279e-6e3a-4104-9934-a6707fa63885" containerName="main" containerID="cri-o://70a7f07cf38054d2c8d71084ab5889d4d58ca26d4cc7efdbd8537986faf3727d" gracePeriod=45 Sep 12 02:10:19.660594 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:19.660548 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f660a6/simulator-w0-f660a6-kserve-779fbfd878-4wgwt" podUID="18093353-67b9-4a8b-9b50-b07ff688f925" containerName="main" containerID="cri-o://20dbe1b095e80f5cbc89d71fee0cd31b5d22d2a70d99a7516832b029c5a6fdb0" gracePeriod=45 Sep 12 02:10:19.727788 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:19.727755 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c273cea1-7278-4513-8432-ccba4c25c999" path="/var/lib/kubelet/pods/c273cea1-7278-4513-8432-ccba4c25c999/volumes" Sep 12 02:10:19.728091 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:19.728077 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc544667-2a18-453f-8e5d-9456f346a6da" path="/var/lib/kubelet/pods/fc544667-2a18-453f-8e5d-9456f346a6da/volumes" Sep 12 02:10:19.757750 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:19.757713 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:19.853316 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:19.853081 2579 generic.go:356] "Generic (PLEG): container finished" podID="18093353-67b9-4a8b-9b50-b07ff688f925" containerID="20dbe1b095e80f5cbc89d71fee0cd31b5d22d2a70d99a7516832b029c5a6fdb0" exitCode=0 Sep 12 02:10:19.853316 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:19.853234 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f660a6/simulator-w0-f660a6-kserve-779fbfd878-4wgwt" event={"ID":"18093353-67b9-4a8b-9b50-b07ff688f925","Type":"ContainerDied","Data":"20dbe1b095e80f5cbc89d71fee0cd31b5d22d2a70d99a7516832b029c5a6fdb0"} Sep 12 02:10:19.855133 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:19.855074 2579 generic.go:356] "Generic (PLEG): container finished" podID="6c25279e-6e3a-4104-9934-a6707fa63885" containerID="70a7f07cf38054d2c8d71084ab5889d4d58ca26d4cc7efdbd8537986faf3727d" exitCode=0 Sep 12 02:10:19.855303 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:19.855239 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f660a6/distinct-2-w0-f660a6-kserve-5d76f79fd5-ksmq5" event={"ID":"6c25279e-6e3a-4104-9934-a6707fa63885","Type":"ContainerDied","Data":"70a7f07cf38054d2c8d71084ab5889d4d58ca26d4cc7efdbd8537986faf3727d"} Sep 12 02:10:19.923885 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:19.923851 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f660a6/distinct-2-w0-f660a6-kserve-5d76f79fd5-ksmq5" Sep 12 02:10:19.957936 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:19.957904 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f660a6/simulator-w0-f660a6-kserve-779fbfd878-4wgwt" Sep 12 02:10:19.980614 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:19.980574 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:20.051432 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:20.051389 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18093353-67b9-4a8b-9b50-b07ff688f925-tmp-dir\") pod \"18093353-67b9-4a8b-9b50-b07ff688f925\" (UID: \"18093353-67b9-4a8b-9b50-b07ff688f925\") " Sep 12 02:10:20.051432 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:20.051439 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c25279e-6e3a-4104-9934-a6707fa63885-tmp-dir\") pod \"6c25279e-6e3a-4104-9934-a6707fa63885\" (UID: \"6c25279e-6e3a-4104-9934-a6707fa63885\") " Sep 12 02:10:20.051718 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:20.051492 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18093353-67b9-4a8b-9b50-b07ff688f925-model-cache\") pod \"18093353-67b9-4a8b-9b50-b07ff688f925\" (UID: \"18093353-67b9-4a8b-9b50-b07ff688f925\") " Sep 12 02:10:20.051718 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:20.051536 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18093353-67b9-4a8b-9b50-b07ff688f925-tls-certs\") pod \"18093353-67b9-4a8b-9b50-b07ff688f925\" (UID: \"18093353-67b9-4a8b-9b50-b07ff688f925\") " Sep 12 02:10:20.051718 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:20.051560 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c25279e-6e3a-4104-9934-a6707fa63885-tls-certs\") pod \"6c25279e-6e3a-4104-9934-a6707fa63885\" (UID: \"6c25279e-6e3a-4104-9934-a6707fa63885\") " Sep 12 02:10:20.051718 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:20.051577 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c25279e-6e3a-4104-9934-a6707fa63885-home\") pod \"6c25279e-6e3a-4104-9934-a6707fa63885\" (UID: \"6c25279e-6e3a-4104-9934-a6707fa63885\") " Sep 12 02:10:20.051718 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:20.051601 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9c4ng\" (UniqueName: \"kubernetes.io/projected/6c25279e-6e3a-4104-9934-a6707fa63885-kube-api-access-9c4ng\") pod \"6c25279e-6e3a-4104-9934-a6707fa63885\" (UID: \"6c25279e-6e3a-4104-9934-a6707fa63885\") " Sep 12 02:10:20.051718 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:20.051647 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c25279e-6e3a-4104-9934-a6707fa63885-dshm\") pod \"6c25279e-6e3a-4104-9934-a6707fa63885\" (UID: \"6c25279e-6e3a-4104-9934-a6707fa63885\") " Sep 12 02:10:20.052017 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:20.051734 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18093353-67b9-4a8b-9b50-b07ff688f925-dshm\") pod \"18093353-67b9-4a8b-9b50-b07ff688f925\" (UID: \"18093353-67b9-4a8b-9b50-b07ff688f925\") " Sep 12 02:10:20.052017 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:20.051765 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c25279e-6e3a-4104-9934-a6707fa63885-model-cache\") pod \"6c25279e-6e3a-4104-9934-a6707fa63885\" (UID: \"6c25279e-6e3a-4104-9934-a6707fa63885\") " Sep 12 02:10:20.052017 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:20.051757 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18093353-67b9-4a8b-9b50-b07ff688f925-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "18093353-67b9-4a8b-9b50-b07ff688f925" (UID: "18093353-67b9-4a8b-9b50-b07ff688f925"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 12 02:10:20.052017 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:20.051816 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7xnkt\" (UniqueName: \"kubernetes.io/projected/18093353-67b9-4a8b-9b50-b07ff688f925-kube-api-access-7xnkt\") pod \"18093353-67b9-4a8b-9b50-b07ff688f925\" (UID: \"18093353-67b9-4a8b-9b50-b07ff688f925\") " Sep 12 02:10:20.052017 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:20.051828 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18093353-67b9-4a8b-9b50-b07ff688f925-model-cache" (OuterVolumeSpecName: "model-cache") pod "18093353-67b9-4a8b-9b50-b07ff688f925" (UID: "18093353-67b9-4a8b-9b50-b07ff688f925"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 12 02:10:20.052017 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:20.051842 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c25279e-6e3a-4104-9934-a6707fa63885-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6c25279e-6e3a-4104-9934-a6707fa63885" (UID: "6c25279e-6e3a-4104-9934-a6707fa63885"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 12 02:10:20.052017 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:20.051854 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18093353-67b9-4a8b-9b50-b07ff688f925-home\") pod \"18093353-67b9-4a8b-9b50-b07ff688f925\" (UID: \"18093353-67b9-4a8b-9b50-b07ff688f925\") " Sep 12 02:10:20.052391 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:20.052069 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c25279e-6e3a-4104-9934-a6707fa63885-home" (OuterVolumeSpecName: "home") pod "6c25279e-6e3a-4104-9934-a6707fa63885" (UID: "6c25279e-6e3a-4104-9934-a6707fa63885"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 12 02:10:20.052391 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:20.052083 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c25279e-6e3a-4104-9934-a6707fa63885-model-cache" (OuterVolumeSpecName: "model-cache") pod "6c25279e-6e3a-4104-9934-a6707fa63885" (UID: "6c25279e-6e3a-4104-9934-a6707fa63885"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 12 02:10:20.052391 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:20.052277 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18093353-67b9-4a8b-9b50-b07ff688f925-home" (OuterVolumeSpecName: "home") pod "18093353-67b9-4a8b-9b50-b07ff688f925" (UID: "18093353-67b9-4a8b-9b50-b07ff688f925"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 12 02:10:20.052618 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:20.052595 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c25279e-6e3a-4104-9934-a6707fa63885-model-cache\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:10:20.052842 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:20.052623 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18093353-67b9-4a8b-9b50-b07ff688f925-home\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:10:20.052842 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:20.052637 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18093353-67b9-4a8b-9b50-b07ff688f925-tmp-dir\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:10:20.052842 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:20.052649 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c25279e-6e3a-4104-9934-a6707fa63885-tmp-dir\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:10:20.052842 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:20.052661 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18093353-67b9-4a8b-9b50-b07ff688f925-model-cache\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:10:20.052842 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:20.052672 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c25279e-6e3a-4104-9934-a6707fa63885-home\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:10:20.054259 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:20.054219 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c25279e-6e3a-4104-9934-a6707fa63885-kube-api-access-9c4ng" (OuterVolumeSpecName: "kube-api-access-9c4ng") pod "6c25279e-6e3a-4104-9934-a6707fa63885" (UID: "6c25279e-6e3a-4104-9934-a6707fa63885"). InnerVolumeSpecName "kube-api-access-9c4ng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 12 02:10:20.054259 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:20.054243 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18093353-67b9-4a8b-9b50-b07ff688f925-dshm" (OuterVolumeSpecName: "dshm") pod "18093353-67b9-4a8b-9b50-b07ff688f925" (UID: "18093353-67b9-4a8b-9b50-b07ff688f925"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 12 02:10:20.054540 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:20.054523 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c25279e-6e3a-4104-9934-a6707fa63885-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6c25279e-6e3a-4104-9934-a6707fa63885" (UID: "6c25279e-6e3a-4104-9934-a6707fa63885"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 12 02:10:20.054637 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:20.054615 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18093353-67b9-4a8b-9b50-b07ff688f925-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "18093353-67b9-4a8b-9b50-b07ff688f925" (UID: "18093353-67b9-4a8b-9b50-b07ff688f925"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 12 02:10:20.054695 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:20.054679 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18093353-67b9-4a8b-9b50-b07ff688f925-kube-api-access-7xnkt" (OuterVolumeSpecName: "kube-api-access-7xnkt") pod "18093353-67b9-4a8b-9b50-b07ff688f925" (UID: "18093353-67b9-4a8b-9b50-b07ff688f925"). InnerVolumeSpecName "kube-api-access-7xnkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 12 02:10:20.054774 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:20.054728 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c25279e-6e3a-4104-9934-a6707fa63885-dshm" (OuterVolumeSpecName: "dshm") pod "6c25279e-6e3a-4104-9934-a6707fa63885" (UID: "6c25279e-6e3a-4104-9934-a6707fa63885"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 12 02:10:20.154010 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:20.153902 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18093353-67b9-4a8b-9b50-b07ff688f925-dshm\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:10:20.154010 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:20.153951 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xnkt\" (UniqueName: \"kubernetes.io/projected/18093353-67b9-4a8b-9b50-b07ff688f925-kube-api-access-7xnkt\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:10:20.154010 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:20.153961 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18093353-67b9-4a8b-9b50-b07ff688f925-tls-certs\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:10:20.154010 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:20.153971 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c25279e-6e3a-4104-9934-a6707fa63885-tls-certs\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:10:20.154010 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:20.153980 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c4ng\" (UniqueName: \"kubernetes.io/projected/6c25279e-6e3a-4104-9934-a6707fa63885-kube-api-access-9c4ng\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:10:20.154010 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:20.153987 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c25279e-6e3a-4104-9934-a6707fa63885-dshm\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:10:20.758421 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:20.758383 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:20.864206 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:20.864167 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f660a6/simulator-w0-f660a6-kserve-779fbfd878-4wgwt" Sep 12 02:10:20.864396 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:20.864207 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f660a6/simulator-w0-f660a6-kserve-779fbfd878-4wgwt" event={"ID":"18093353-67b9-4a8b-9b50-b07ff688f925","Type":"ContainerDied","Data":"20f60701034ac0d9ca9396071acbc2adbefe819fff50b2a217e98bb0399254a1"} Sep 12 02:10:20.864396 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:20.864261 2579 scope.go:117] "RemoveContainer" containerID="20dbe1b095e80f5cbc89d71fee0cd31b5d22d2a70d99a7516832b029c5a6fdb0" Sep 12 02:10:20.865714 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:20.865690 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f660a6/distinct-2-w0-f660a6-kserve-5d76f79fd5-ksmq5" event={"ID":"6c25279e-6e3a-4104-9934-a6707fa63885","Type":"ContainerDied","Data":"ae89eeeb61eb6f131d6fdce87ea22b7a05506161e55060bdb59a428d151c03b8"} Sep 12 02:10:20.865791 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:20.865734 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f660a6/distinct-2-w0-f660a6-kserve-5d76f79fd5-ksmq5" Sep 12 02:10:20.875183 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:20.875164 2579 scope.go:117] "RemoveContainer" containerID="70a7f07cf38054d2c8d71084ab5889d4d58ca26d4cc7efdbd8537986faf3727d" Sep 12 02:10:20.887763 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:20.887733 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f660a6/simulator-w0-f660a6-kserve-779fbfd878-4wgwt"] Sep 12 02:10:20.891599 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:20.891534 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f660a6/simulator-w0-f660a6-kserve-779fbfd878-4wgwt"] Sep 12 02:10:20.903231 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:20.903199 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f660a6/distinct-2-w0-f660a6-kserve-5d76f79fd5-ksmq5"] Sep 12 02:10:20.908117 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:20.908091 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f660a6/distinct-2-w0-f660a6-kserve-5d76f79fd5-ksmq5"] Sep 12 02:10:21.397954 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:21.397915 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:21.727392 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:21.727310 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18093353-67b9-4a8b-9b50-b07ff688f925" path="/var/lib/kubelet/pods/18093353-67b9-4a8b-9b50-b07ff688f925/volumes" Sep 12 02:10:21.727751 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:21.727736 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c25279e-6e3a-4104-9934-a6707fa63885" path="/var/lib/kubelet/pods/6c25279e-6e3a-4104-9934-a6707fa63885/volumes" Sep 12 02:10:21.832535 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:21.832505 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b23bee-774f64cf68-hsjbn" Sep 12 02:10:22.198430 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:22.198395 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:23.091441 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:23.091403 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:23.870941 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:23.870906 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:24.671596 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:24.671557 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:25.953985 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:25.953945 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:26.055594 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:26.055555 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:26.154737 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:26.154706 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:26.228443 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:26.228362 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:26.829847 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:26.829815 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b23bee-898989fc4-kz7qg" Sep 12 02:10:26.947330 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:26.947286 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:27.594215 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:27.594180 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:28.249096 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:28.249058 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:28.355179 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:28.355146 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:29.162042 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:29.161987 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:29.800680 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:29.800643 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:30.599920 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:30.599875 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:31.263800 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:31.263766 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:32.053984 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:32.053924 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:32.732087 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:32.732052 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:33.560038 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:33.559999 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:34.277864 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:34.277826 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:35.071390 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:35.071362 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:35.859067 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:35.859030 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:36.601585 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:36.601549 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:37.238471 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:37.238416 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:37.959893 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:37.959858 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:41.070481 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:41.070422 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:41.169759 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:41.169712 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:41.372230 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:41.372171 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:42.071859 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:42.071820 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:47.020564 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:47.020524 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:47.063098 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:47.063060 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-caecee-659b9c48cd-tvr4c"] Sep 12 02:10:47.063376 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:47.063340 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-caecee-659b9c48cd-tvr4c" podUID="132fc085-122b-48ab-b07e-dd81c6c1c516" containerName="payload-pre-processing" containerID="cri-o://a2d5561aaccc545041e1846c7225d9dd098069d795884594755f0c0e135e9790" gracePeriod=30 Sep 12 02:10:47.121637 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:47.121598 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:47.219912 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:47.219878 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:47.292668 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:47.292540 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:47.329731 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:47.329702 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-caecee-659b9c48cd-tvr4c" Sep 12 02:10:47.444298 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:47.444263 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/132fc085-122b-48ab-b07e-dd81c6c1c516-plugins-config-volume\") pod \"132fc085-122b-48ab-b07e-dd81c6c1c516\" (UID: \"132fc085-122b-48ab-b07e-dd81c6c1c516\") " Sep 12 02:10:47.444298 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:47.444318 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bkg9d\" (UniqueName: \"kubernetes.io/projected/132fc085-122b-48ab-b07e-dd81c6c1c516-kube-api-access-bkg9d\") pod \"132fc085-122b-48ab-b07e-dd81c6c1c516\" (UID: \"132fc085-122b-48ab-b07e-dd81c6c1c516\") " Sep 12 02:10:47.444746 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:47.444720 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/132fc085-122b-48ab-b07e-dd81c6c1c516-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "132fc085-122b-48ab-b07e-dd81c6c1c516" (UID: "132fc085-122b-48ab-b07e-dd81c6c1c516"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 12 02:10:47.446736 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:47.446712 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/132fc085-122b-48ab-b07e-dd81c6c1c516-kube-api-access-bkg9d" (OuterVolumeSpecName: "kube-api-access-bkg9d") pod "132fc085-122b-48ab-b07e-dd81c6c1c516" (UID: "132fc085-122b-48ab-b07e-dd81c6c1c516"). InnerVolumeSpecName "kube-api-access-bkg9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 12 02:10:47.545429 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:47.545336 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/132fc085-122b-48ab-b07e-dd81c6c1c516-plugins-config-volume\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:10:47.545429 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:47.545378 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkg9d\" (UniqueName: \"kubernetes.io/projected/132fc085-122b-48ab-b07e-dd81c6c1c516-kube-api-access-bkg9d\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:10:47.636756 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:47.636715 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:47.737478 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:47.737431 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:48.023378 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:48.023351 2579 generic.go:356] "Generic (PLEG): container finished" podID="132fc085-122b-48ab-b07e-dd81c6c1c516" containerID="a2d5561aaccc545041e1846c7225d9dd098069d795884594755f0c0e135e9790" exitCode=0 Sep 12 02:10:48.023791 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:48.023402 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-caecee-659b9c48cd-tvr4c" Sep 12 02:10:48.023791 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:48.023433 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-caecee-659b9c48cd-tvr4c" event={"ID":"132fc085-122b-48ab-b07e-dd81c6c1c516","Type":"ContainerDied","Data":"a2d5561aaccc545041e1846c7225d9dd098069d795884594755f0c0e135e9790"} Sep 12 02:10:48.023791 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:48.023521 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-caecee-659b9c48cd-tvr4c" event={"ID":"132fc085-122b-48ab-b07e-dd81c6c1c516","Type":"ContainerDied","Data":"7edcbfe5791413426d1c0133337f7b64daf9f441677db60a85d9e2e7bef976fd"} Sep 12 02:10:48.023791 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:48.023550 2579 scope.go:117] "RemoveContainer" containerID="a2d5561aaccc545041e1846c7225d9dd098069d795884594755f0c0e135e9790" Sep 12 02:10:48.033055 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:48.033034 2579 scope.go:117] "RemoveContainer" containerID="a2d5561aaccc545041e1846c7225d9dd098069d795884594755f0c0e135e9790" Sep 12 02:10:48.033375 ip-10-0-133-79 kubenswrapper[2579]: E0912 02:10:48.033354 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2d5561aaccc545041e1846c7225d9dd098069d795884594755f0c0e135e9790\": container with ID starting with a2d5561aaccc545041e1846c7225d9dd098069d795884594755f0c0e135e9790 not found: ID does not exist" containerID="a2d5561aaccc545041e1846c7225d9dd098069d795884594755f0c0e135e9790" Sep 12 02:10:48.033478 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:48.033384 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2d5561aaccc545041e1846c7225d9dd098069d795884594755f0c0e135e9790"} err="failed to get container status \"a2d5561aaccc545041e1846c7225d9dd098069d795884594755f0c0e135e9790\": rpc error: code = NotFound desc = could not find container \"a2d5561aaccc545041e1846c7225d9dd098069d795884594755f0c0e135e9790\": container with ID starting with a2d5561aaccc545041e1846c7225d9dd098069d795884594755f0c0e135e9790 not found: ID does not exist" Sep 12 02:10:48.039594 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:48.039570 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-caecee-659b9c48cd-tvr4c"] Sep 12 02:10:48.043650 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:48.043618 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-caecee-659b9c48cd-tvr4c"] Sep 12 02:10:48.340018 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:48.339925 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:48.536142 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:48.536107 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:49.059488 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:49.059434 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:49.727897 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:49.727864 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="132fc085-122b-48ab-b07e-dd81c6c1c516" path="/var/lib/kubelet/pods/132fc085-122b-48ab-b07e-dd81c6c1c516/volumes" Sep 12 02:10:49.764312 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:49.764279 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:50.420960 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:50.420915 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:51.166090 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:51.166046 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:51.237979 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:51.237943 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:10:51.826023 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:51.825986 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:52.475610 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:52.475567 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:54.352109 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:54.352072 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:54.452225 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:54.452189 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:54.635299 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:54.635269 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:55.077656 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:55.077567 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:55.866613 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:55.866580 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:56.859170 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:56.859133 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:57.658725 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:57.658693 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:57.941874 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:57.941778 2579 scope.go:117] "RemoveContainer" containerID="84d3ed5fec90d2131378fcaba499e7eb57219bfadaac578e979b89813a1f7ef4" Sep 12 02:10:58.301439 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:58.301350 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:59.042835 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:59.042799 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:10:59.676686 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:10:59.676658 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:00.319358 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:00.319320 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:10.594003 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:10.593967 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b23bee-774f64cf68-hsjbn"] Sep 12 02:11:10.594495 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:10.594242 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b23bee-774f64cf68-hsjbn" podUID="9c9f8ad4-58c8-45a9-8183-00a5976e1f54" containerName="maas-api" containerID="cri-o://296d5d296e41ad934b0289c2c127a68dab587b14344d69250312545d086c1f45" gracePeriod=30 Sep 12 02:11:10.627106 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:10.627075 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:10.655653 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:10.655617 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b23bee-898989fc4-kz7qg"] Sep 12 02:11:10.655902 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:10.655871 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b23bee-898989fc4-kz7qg" podUID="3f1890ff-65c0-4771-9d6b-6887073fa37f" containerName="payload-pre-processing" containerID="cri-o://96580acd2335a93fec7984926895ebba7678ad7074f20890982b2c58baad6375" gracePeriod=30 Sep 12 02:11:10.725939 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:10.725903 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:10.807824 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:10.807788 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:10.870137 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:10.870045 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:10.909255 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:10.909226 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b23bee-774f64cf68-hsjbn" Sep 12 02:11:10.944650 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:10.944626 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b23bee-898989fc4-kz7qg" Sep 12 02:11:10.984655 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:10.984623 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c9f8ad4-58c8-45a9-8183-00a5976e1f54-metrics-certs\") pod \"9c9f8ad4-58c8-45a9-8183-00a5976e1f54\" (UID: \"9c9f8ad4-58c8-45a9-8183-00a5976e1f54\") " Sep 12 02:11:10.984822 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:10.984671 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c9f8ad4-58c8-45a9-8183-00a5976e1f54-maas-api-tls\") pod \"9c9f8ad4-58c8-45a9-8183-00a5976e1f54\" (UID: \"9c9f8ad4-58c8-45a9-8183-00a5976e1f54\") " Sep 12 02:11:10.984822 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:10.984720 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jsmk6\" (UniqueName: \"kubernetes.io/projected/9c9f8ad4-58c8-45a9-8183-00a5976e1f54-kube-api-access-jsmk6\") pod \"9c9f8ad4-58c8-45a9-8183-00a5976e1f54\" (UID: \"9c9f8ad4-58c8-45a9-8183-00a5976e1f54\") " Sep 12 02:11:10.984822 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:10.984781 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c9f8ad4-58c8-45a9-8183-00a5976e1f54-openshift-service-ca\") pod \"9c9f8ad4-58c8-45a9-8183-00a5976e1f54\" (UID: \"9c9f8ad4-58c8-45a9-8183-00a5976e1f54\") " Sep 12 02:11:10.984822 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:10.984804 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f1890ff-65c0-4771-9d6b-6887073fa37f-plugins-config-volume\") pod \"3f1890ff-65c0-4771-9d6b-6887073fa37f\" (UID: \"3f1890ff-65c0-4771-9d6b-6887073fa37f\") " Sep 12 02:11:10.985031 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:10.984884 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mvm4w\" (UniqueName: \"kubernetes.io/projected/3f1890ff-65c0-4771-9d6b-6887073fa37f-kube-api-access-mvm4w\") pod \"3f1890ff-65c0-4771-9d6b-6887073fa37f\" (UID: \"3f1890ff-65c0-4771-9d6b-6887073fa37f\") " Sep 12 02:11:10.985296 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:10.985149 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c9f8ad4-58c8-45a9-8183-00a5976e1f54-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9c9f8ad4-58c8-45a9-8183-00a5976e1f54" (UID: "9c9f8ad4-58c8-45a9-8183-00a5976e1f54"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 12 02:11:10.985296 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:10.985238 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f1890ff-65c0-4771-9d6b-6887073fa37f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3f1890ff-65c0-4771-9d6b-6887073fa37f" (UID: "3f1890ff-65c0-4771-9d6b-6887073fa37f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 12 02:11:10.985296 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:10.985289 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c9f8ad4-58c8-45a9-8183-00a5976e1f54-openshift-service-ca\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:11:10.987304 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:10.987281 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f1890ff-65c0-4771-9d6b-6887073fa37f-kube-api-access-mvm4w" (OuterVolumeSpecName: "kube-api-access-mvm4w") pod "3f1890ff-65c0-4771-9d6b-6887073fa37f" (UID: "3f1890ff-65c0-4771-9d6b-6887073fa37f"). InnerVolumeSpecName "kube-api-access-mvm4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 12 02:11:10.987589 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:10.987564 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c9f8ad4-58c8-45a9-8183-00a5976e1f54-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9c9f8ad4-58c8-45a9-8183-00a5976e1f54" (UID: "9c9f8ad4-58c8-45a9-8183-00a5976e1f54"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 12 02:11:10.987589 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:10.987582 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c9f8ad4-58c8-45a9-8183-00a5976e1f54-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9c9f8ad4-58c8-45a9-8183-00a5976e1f54" (UID: "9c9f8ad4-58c8-45a9-8183-00a5976e1f54"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 12 02:11:10.987707 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:10.987611 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c9f8ad4-58c8-45a9-8183-00a5976e1f54-kube-api-access-jsmk6" (OuterVolumeSpecName: "kube-api-access-jsmk6") pod "9c9f8ad4-58c8-45a9-8183-00a5976e1f54" (UID: "9c9f8ad4-58c8-45a9-8183-00a5976e1f54"). InnerVolumeSpecName "kube-api-access-jsmk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 12 02:11:11.086781 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:11.086735 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsmk6\" (UniqueName: \"kubernetes.io/projected/9c9f8ad4-58c8-45a9-8183-00a5976e1f54-kube-api-access-jsmk6\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:11:11.086781 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:11.086772 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f1890ff-65c0-4771-9d6b-6887073fa37f-plugins-config-volume\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:11:11.086781 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:11.086785 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvm4w\" (UniqueName: \"kubernetes.io/projected/3f1890ff-65c0-4771-9d6b-6887073fa37f-kube-api-access-mvm4w\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:11:11.087032 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:11.086796 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c9f8ad4-58c8-45a9-8183-00a5976e1f54-metrics-certs\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:11:11.087032 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:11.086810 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c9f8ad4-58c8-45a9-8183-00a5976e1f54-maas-api-tls\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:11:11.143322 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:11.143285 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:11.163492 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:11.163428 2579 generic.go:356] "Generic (PLEG): container finished" podID="3f1890ff-65c0-4771-9d6b-6887073fa37f" containerID="96580acd2335a93fec7984926895ebba7678ad7074f20890982b2c58baad6375" exitCode=0 Sep 12 02:11:11.163668 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:11.163506 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b23bee-898989fc4-kz7qg" Sep 12 02:11:11.163668 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:11.163514 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b23bee-898989fc4-kz7qg" event={"ID":"3f1890ff-65c0-4771-9d6b-6887073fa37f","Type":"ContainerDied","Data":"96580acd2335a93fec7984926895ebba7678ad7074f20890982b2c58baad6375"} Sep 12 02:11:11.163668 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:11.163568 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b23bee-898989fc4-kz7qg" event={"ID":"3f1890ff-65c0-4771-9d6b-6887073fa37f","Type":"ContainerDied","Data":"c2b87cfc0a47fe1a1cb382faa913715b8f96f2799eba28bf27874241e53cd3bc"} Sep 12 02:11:11.163668 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:11.163590 2579 scope.go:117] "RemoveContainer" containerID="96580acd2335a93fec7984926895ebba7678ad7074f20890982b2c58baad6375" Sep 12 02:11:11.164965 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:11.164943 2579 generic.go:356] "Generic (PLEG): container finished" podID="9c9f8ad4-58c8-45a9-8183-00a5976e1f54" containerID="296d5d296e41ad934b0289c2c127a68dab587b14344d69250312545d086c1f45" exitCode=0 Sep 12 02:11:11.165074 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:11.164994 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b23bee-774f64cf68-hsjbn" Sep 12 02:11:11.165074 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:11.164994 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b23bee-774f64cf68-hsjbn" event={"ID":"9c9f8ad4-58c8-45a9-8183-00a5976e1f54","Type":"ContainerDied","Data":"296d5d296e41ad934b0289c2c127a68dab587b14344d69250312545d086c1f45"} Sep 12 02:11:11.165074 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:11.165055 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b23bee-774f64cf68-hsjbn" event={"ID":"9c9f8ad4-58c8-45a9-8183-00a5976e1f54","Type":"ContainerDied","Data":"13119859bc3c0f3a2cbb6bd8c1d42059a8c8a10ceca6a3fcc95e97525c1b7bb8"} Sep 12 02:11:11.174378 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:11.174362 2579 scope.go:117] "RemoveContainer" containerID="96580acd2335a93fec7984926895ebba7678ad7074f20890982b2c58baad6375" Sep 12 02:11:11.174719 ip-10-0-133-79 kubenswrapper[2579]: E0912 02:11:11.174701 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96580acd2335a93fec7984926895ebba7678ad7074f20890982b2c58baad6375\": container with ID starting with 96580acd2335a93fec7984926895ebba7678ad7074f20890982b2c58baad6375 not found: ID does not exist" containerID="96580acd2335a93fec7984926895ebba7678ad7074f20890982b2c58baad6375" Sep 12 02:11:11.174819 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:11.174729 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96580acd2335a93fec7984926895ebba7678ad7074f20890982b2c58baad6375"} err="failed to get container status \"96580acd2335a93fec7984926895ebba7678ad7074f20890982b2c58baad6375\": rpc error: code = NotFound desc = could not find container \"96580acd2335a93fec7984926895ebba7678ad7074f20890982b2c58baad6375\": container with ID starting with 96580acd2335a93fec7984926895ebba7678ad7074f20890982b2c58baad6375 not found: ID does not exist" Sep 12 02:11:11.174819 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:11.174746 2579 scope.go:117] "RemoveContainer" containerID="296d5d296e41ad934b0289c2c127a68dab587b14344d69250312545d086c1f45" Sep 12 02:11:11.183848 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:11.183830 2579 scope.go:117] "RemoveContainer" containerID="296d5d296e41ad934b0289c2c127a68dab587b14344d69250312545d086c1f45" Sep 12 02:11:11.184123 ip-10-0-133-79 kubenswrapper[2579]: E0912 02:11:11.184104 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"296d5d296e41ad934b0289c2c127a68dab587b14344d69250312545d086c1f45\": container with ID starting with 296d5d296e41ad934b0289c2c127a68dab587b14344d69250312545d086c1f45 not found: ID does not exist" containerID="296d5d296e41ad934b0289c2c127a68dab587b14344d69250312545d086c1f45" Sep 12 02:11:11.184187 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:11.184131 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"296d5d296e41ad934b0289c2c127a68dab587b14344d69250312545d086c1f45"} err="failed to get container status \"296d5d296e41ad934b0289c2c127a68dab587b14344d69250312545d086c1f45\": rpc error: code = NotFound desc = could not find container \"296d5d296e41ad934b0289c2c127a68dab587b14344d69250312545d086c1f45\": container with ID starting with 296d5d296e41ad934b0289c2c127a68dab587b14344d69250312545d086c1f45 not found: ID does not exist" Sep 12 02:11:11.190854 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:11.190829 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b23bee-898989fc4-kz7qg"] Sep 12 02:11:11.192970 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:11.192951 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b23bee-898989fc4-kz7qg"] Sep 12 02:11:11.209310 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:11.209273 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b23bee-774f64cf68-hsjbn"] Sep 12 02:11:11.214752 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:11.214724 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b23bee-774f64cf68-hsjbn"] Sep 12 02:11:11.226796 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:11.226764 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:11.727913 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:11.727879 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f1890ff-65c0-4771-9d6b-6887073fa37f" path="/var/lib/kubelet/pods/3f1890ff-65c0-4771-9d6b-6887073fa37f/volumes" Sep 12 02:11:11.728312 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:11.728297 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c9f8ad4-58c8-45a9-8183-00a5976e1f54" path="/var/lib/kubelet/pods/9c9f8ad4-58c8-45a9-8183-00a5976e1f54/volumes" Sep 12 02:11:11.966312 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:11.966276 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:12.696903 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:12.696840 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:13.327529 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:13.327489 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:14.060276 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:14.060240 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:14.681566 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:14.681525 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:17.252527 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:17.252489 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:17.366674 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:17.366640 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:17.542046 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:17.541951 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:18.141216 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:18.141177 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:18.955207 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:18.955173 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:19.573876 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:19.573836 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:19.625977 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:19.625942 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:11:20.197210 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:20.197169 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:20.862937 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:20.862898 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:21.535362 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:21.535321 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:22.264240 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:22.264203 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:26.591675 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:26.591609 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:26.692288 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:26.692255 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:27.457607 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:27.457565 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:28.144860 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:28.144824 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:28.680031 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:28.679999 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e8046f-66dc7c9f5d-vc894"] Sep 12 02:11:28.680319 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:28.680270 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e8046f-66dc7c9f5d-vc894" podUID="1c105641-487e-4a89-a4d5-a4c2b9206411" containerName="maas-api" containerID="cri-o://d4fede751a7fc892d506dbfd27571a82d88e83f4e8ff7ed791d00a12367d2b73" gracePeriod=30 Sep 12 02:11:28.708067 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:28.708037 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:28.736644 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:28.736609 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e8046f-55844c5498-pcpbr"] Sep 12 02:11:28.736904 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:28.736876 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e8046f-55844c5498-pcpbr" podUID="a007efd1-b179-445a-84ea-28cf8f744c9b" containerName="payload-pre-processing" containerID="cri-o://d389cd9b85386df3a3bd39a9cf664c6d0ecb4e3e63a9c185d0f794eea9a99cae" gracePeriod=30 Sep 12 02:11:28.778673 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:28.778637 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:28.878581 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:28.878544 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:28.940806 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:28.940663 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:28.991799 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:28.991761 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e8046f-66dc7c9f5d-vc894" Sep 12 02:11:29.026758 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:29.026729 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e8046f-55844c5498-pcpbr" Sep 12 02:11:29.038051 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:29.038003 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:29.071357 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:29.071324 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xs4gg\" (UniqueName: \"kubernetes.io/projected/1c105641-487e-4a89-a4d5-a4c2b9206411-kube-api-access-xs4gg\") pod \"1c105641-487e-4a89-a4d5-a4c2b9206411\" (UID: \"1c105641-487e-4a89-a4d5-a4c2b9206411\") " Sep 12 02:11:29.071583 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:29.071444 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c105641-487e-4a89-a4d5-a4c2b9206411-openshift-service-ca\") pod \"1c105641-487e-4a89-a4d5-a4c2b9206411\" (UID: \"1c105641-487e-4a89-a4d5-a4c2b9206411\") " Sep 12 02:11:29.071583 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:29.071519 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c105641-487e-4a89-a4d5-a4c2b9206411-maas-api-tls\") pod \"1c105641-487e-4a89-a4d5-a4c2b9206411\" (UID: \"1c105641-487e-4a89-a4d5-a4c2b9206411\") " Sep 12 02:11:29.071583 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:29.071561 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c105641-487e-4a89-a4d5-a4c2b9206411-metrics-certs\") pod \"1c105641-487e-4a89-a4d5-a4c2b9206411\" (UID: \"1c105641-487e-4a89-a4d5-a4c2b9206411\") " Sep 12 02:11:29.071871 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:29.071847 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c105641-487e-4a89-a4d5-a4c2b9206411-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1c105641-487e-4a89-a4d5-a4c2b9206411" (UID: "1c105641-487e-4a89-a4d5-a4c2b9206411"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 12 02:11:29.073718 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:29.073688 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c105641-487e-4a89-a4d5-a4c2b9206411-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1c105641-487e-4a89-a4d5-a4c2b9206411" (UID: "1c105641-487e-4a89-a4d5-a4c2b9206411"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 12 02:11:29.073718 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:29.073688 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c105641-487e-4a89-a4d5-a4c2b9206411-kube-api-access-xs4gg" (OuterVolumeSpecName: "kube-api-access-xs4gg") pod "1c105641-487e-4a89-a4d5-a4c2b9206411" (UID: "1c105641-487e-4a89-a4d5-a4c2b9206411"). InnerVolumeSpecName "kube-api-access-xs4gg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 12 02:11:29.073862 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:29.073804 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c105641-487e-4a89-a4d5-a4c2b9206411-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1c105641-487e-4a89-a4d5-a4c2b9206411" (UID: "1c105641-487e-4a89-a4d5-a4c2b9206411"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 12 02:11:29.172827 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:29.172792 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jjvr9\" (UniqueName: \"kubernetes.io/projected/a007efd1-b179-445a-84ea-28cf8f744c9b-kube-api-access-jjvr9\") pod \"a007efd1-b179-445a-84ea-28cf8f744c9b\" (UID: \"a007efd1-b179-445a-84ea-28cf8f744c9b\") " Sep 12 02:11:29.173230 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:29.172916 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a007efd1-b179-445a-84ea-28cf8f744c9b-plugins-config-volume\") pod \"a007efd1-b179-445a-84ea-28cf8f744c9b\" (UID: \"a007efd1-b179-445a-84ea-28cf8f744c9b\") " Sep 12 02:11:29.173301 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:29.173281 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c105641-487e-4a89-a4d5-a4c2b9206411-openshift-service-ca\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:11:29.173341 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:29.173308 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c105641-487e-4a89-a4d5-a4c2b9206411-maas-api-tls\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:11:29.173341 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:29.173324 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c105641-487e-4a89-a4d5-a4c2b9206411-metrics-certs\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:11:29.173341 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:29.173321 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a007efd1-b179-445a-84ea-28cf8f744c9b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a007efd1-b179-445a-84ea-28cf8f744c9b" (UID: "a007efd1-b179-445a-84ea-28cf8f744c9b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 12 02:11:29.173341 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:29.173337 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs4gg\" (UniqueName: \"kubernetes.io/projected/1c105641-487e-4a89-a4d5-a4c2b9206411-kube-api-access-xs4gg\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:11:29.174999 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:29.174976 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a007efd1-b179-445a-84ea-28cf8f744c9b-kube-api-access-jjvr9" (OuterVolumeSpecName: "kube-api-access-jjvr9") pod "a007efd1-b179-445a-84ea-28cf8f744c9b" (UID: "a007efd1-b179-445a-84ea-28cf8f744c9b"). InnerVolumeSpecName "kube-api-access-jjvr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 12 02:11:29.266886 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:29.266771 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:29.267306 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:29.267284 2579 generic.go:356] "Generic (PLEG): container finished" podID="a007efd1-b179-445a-84ea-28cf8f744c9b" containerID="d389cd9b85386df3a3bd39a9cf664c6d0ecb4e3e63a9c185d0f794eea9a99cae" exitCode=0 Sep 12 02:11:29.267405 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:29.267356 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e8046f-55844c5498-pcpbr" Sep 12 02:11:29.267405 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:29.267364 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e8046f-55844c5498-pcpbr" event={"ID":"a007efd1-b179-445a-84ea-28cf8f744c9b","Type":"ContainerDied","Data":"d389cd9b85386df3a3bd39a9cf664c6d0ecb4e3e63a9c185d0f794eea9a99cae"} Sep 12 02:11:29.267558 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:29.267417 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e8046f-55844c5498-pcpbr" event={"ID":"a007efd1-b179-445a-84ea-28cf8f744c9b","Type":"ContainerDied","Data":"2f7bb4ca668a54cafc85c7af68c7156578033d291c7cfc56772772cbf9142be2"} Sep 12 02:11:29.267558 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:29.267442 2579 scope.go:117] "RemoveContainer" containerID="d389cd9b85386df3a3bd39a9cf664c6d0ecb4e3e63a9c185d0f794eea9a99cae" Sep 12 02:11:29.269016 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:29.268986 2579 generic.go:356] "Generic (PLEG): container finished" podID="1c105641-487e-4a89-a4d5-a4c2b9206411" containerID="d4fede751a7fc892d506dbfd27571a82d88e83f4e8ff7ed791d00a12367d2b73" exitCode=0 Sep 12 02:11:29.269141 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:29.269012 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e8046f-66dc7c9f5d-vc894" event={"ID":"1c105641-487e-4a89-a4d5-a4c2b9206411","Type":"ContainerDied","Data":"d4fede751a7fc892d506dbfd27571a82d88e83f4e8ff7ed791d00a12367d2b73"} Sep 12 02:11:29.269141 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:29.269038 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e8046f-66dc7c9f5d-vc894" Sep 12 02:11:29.269141 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:29.269075 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e8046f-66dc7c9f5d-vc894" event={"ID":"1c105641-487e-4a89-a4d5-a4c2b9206411","Type":"ContainerDied","Data":"525f2579b5a4608264e14598b8643147ee326bcea1fdc65c1f8b2b0c48ccd030"} Sep 12 02:11:29.273989 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:29.273968 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjvr9\" (UniqueName: \"kubernetes.io/projected/a007efd1-b179-445a-84ea-28cf8f744c9b-kube-api-access-jjvr9\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:11:29.273989 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:29.273990 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a007efd1-b179-445a-84ea-28cf8f744c9b-plugins-config-volume\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:11:29.278703 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:29.278687 2579 scope.go:117] "RemoveContainer" containerID="d389cd9b85386df3a3bd39a9cf664c6d0ecb4e3e63a9c185d0f794eea9a99cae" Sep 12 02:11:29.279018 ip-10-0-133-79 kubenswrapper[2579]: E0912 02:11:29.278993 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d389cd9b85386df3a3bd39a9cf664c6d0ecb4e3e63a9c185d0f794eea9a99cae\": container with ID starting with d389cd9b85386df3a3bd39a9cf664c6d0ecb4e3e63a9c185d0f794eea9a99cae not found: ID does not exist" containerID="d389cd9b85386df3a3bd39a9cf664c6d0ecb4e3e63a9c185d0f794eea9a99cae" Sep 12 02:11:29.279101 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:29.279032 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d389cd9b85386df3a3bd39a9cf664c6d0ecb4e3e63a9c185d0f794eea9a99cae"} err="failed to get container status \"d389cd9b85386df3a3bd39a9cf664c6d0ecb4e3e63a9c185d0f794eea9a99cae\": rpc error: code = NotFound desc = could not find container \"d389cd9b85386df3a3bd39a9cf664c6d0ecb4e3e63a9c185d0f794eea9a99cae\": container with ID starting with d389cd9b85386df3a3bd39a9cf664c6d0ecb4e3e63a9c185d0f794eea9a99cae not found: ID does not exist" Sep 12 02:11:29.279101 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:29.279056 2579 scope.go:117] "RemoveContainer" containerID="d4fede751a7fc892d506dbfd27571a82d88e83f4e8ff7ed791d00a12367d2b73" Sep 12 02:11:29.288403 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:29.288384 2579 scope.go:117] "RemoveContainer" containerID="d4fede751a7fc892d506dbfd27571a82d88e83f4e8ff7ed791d00a12367d2b73" Sep 12 02:11:29.288757 ip-10-0-133-79 kubenswrapper[2579]: E0912 02:11:29.288737 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4fede751a7fc892d506dbfd27571a82d88e83f4e8ff7ed791d00a12367d2b73\": container with ID starting with d4fede751a7fc892d506dbfd27571a82d88e83f4e8ff7ed791d00a12367d2b73 not found: ID does not exist" containerID="d4fede751a7fc892d506dbfd27571a82d88e83f4e8ff7ed791d00a12367d2b73" Sep 12 02:11:29.288812 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:29.288766 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4fede751a7fc892d506dbfd27571a82d88e83f4e8ff7ed791d00a12367d2b73"} err="failed to get container status \"d4fede751a7fc892d506dbfd27571a82d88e83f4e8ff7ed791d00a12367d2b73\": rpc error: code = NotFound desc = could not find container \"d4fede751a7fc892d506dbfd27571a82d88e83f4e8ff7ed791d00a12367d2b73\": container with ID starting with d4fede751a7fc892d506dbfd27571a82d88e83f4e8ff7ed791d00a12367d2b73 not found: ID does not exist" Sep 12 02:11:29.293743 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:29.293717 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e8046f-55844c5498-pcpbr"] Sep 12 02:11:29.297252 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:29.297227 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e8046f-55844c5498-pcpbr"] Sep 12 02:11:29.307317 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:29.307285 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e8046f-66dc7c9f5d-vc894"] Sep 12 02:11:29.311247 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:29.311209 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e8046f-66dc7c9f5d-vc894"] Sep 12 02:11:29.726364 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:29.726319 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c105641-487e-4a89-a4d5-a4c2b9206411" path="/var/lib/kubelet/pods/1c105641-487e-4a89-a4d5-a4c2b9206411/volumes" Sep 12 02:11:29.726833 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:29.726812 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a007efd1-b179-445a-84ea-28cf8f744c9b" path="/var/lib/kubelet/pods/a007efd1-b179-445a-84ea-28cf8f744c9b/volumes" Sep 12 02:11:29.894445 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:29.894411 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:30.660634 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:30.660598 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:31.285242 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:31.285208 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:31.640971 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:31.640932 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-98f68e/distinct-2-w4-98f68e-kserve-c5758bc66-d9c6h"] Sep 12 02:11:31.719050 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:31.719018 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:32.171852 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:32.171811 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:32.863153 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:32.863115 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:33.100098 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:33.100058 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:35.354502 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:35.354438 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:35.458004 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:35.457917 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:35.655924 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:35.655883 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:36.858766 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:36.858736 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:37.093388 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:37.093349 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:37.502590 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:37.502565 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:38.362580 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:38.362541 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:39.154021 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:39.153980 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:44.025756 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:44.025719 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:44.123380 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:44.123337 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:44.129330 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:44.129305 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:11:44.221603 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:44.221564 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:44.278650 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:44.278551 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:44.569101 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:44.569008 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:45.168434 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:45.168399 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:45.859763 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:45.859728 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:46.662028 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:46.661989 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-98f68e/distinct-2-w4-98f68e-kserve-c5758bc66-d9c6h" podUID="cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5" containerName="main" containerID="cri-o://686395bfb451da8345b0f20de213d06db6f065515b7346eb03a99e5d99c24647" gracePeriod=45 Sep 12 02:11:46.774329 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:46.774293 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:46.916197 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:46.916126 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-98f68e/distinct-2-w4-98f68e-kserve-c5758bc66-d9c6h" Sep 12 02:11:46.950158 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:46.950125 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rs92k\" (UniqueName: \"kubernetes.io/projected/cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5-kube-api-access-rs92k\") pod \"cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5\" (UID: \"cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5\") " Sep 12 02:11:46.950331 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:46.950227 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5-tmp-dir\") pod \"cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5\" (UID: \"cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5\") " Sep 12 02:11:46.950331 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:46.950297 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5-home\") pod \"cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5\" (UID: \"cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5\") " Sep 12 02:11:46.950331 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:46.950325 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5-dshm\") pod \"cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5\" (UID: \"cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5\") " Sep 12 02:11:46.950538 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:46.950358 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5-model-cache\") pod \"cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5\" (UID: \"cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5\") " Sep 12 02:11:46.950538 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:46.950398 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5-tls-certs\") pod \"cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5\" (UID: \"cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5\") " Sep 12 02:11:46.950671 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:46.950645 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5-home" (OuterVolumeSpecName: "home") pod "cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5" (UID: "cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 12 02:11:46.950737 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:46.950671 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5-model-cache" (OuterVolumeSpecName: "model-cache") pod "cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5" (UID: "cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 12 02:11:46.950845 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:46.950822 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5" (UID: "cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 12 02:11:46.950933 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:46.950833 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5-home\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:11:46.950933 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:46.950870 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5-model-cache\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:11:46.952932 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:46.952907 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5" (UID: "cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 12 02:11:46.953074 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:46.953055 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5-kube-api-access-rs92k" (OuterVolumeSpecName: "kube-api-access-rs92k") pod "cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5" (UID: "cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5"). InnerVolumeSpecName "kube-api-access-rs92k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 12 02:11:46.953074 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:46.953059 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5-dshm" (OuterVolumeSpecName: "dshm") pod "cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5" (UID: "cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 12 02:11:47.052208 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:47.052161 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5-tmp-dir\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:11:47.052208 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:47.052210 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5-dshm\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:11:47.052412 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:47.052224 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5-tls-certs\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:11:47.052412 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:47.052240 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs92k\" (UniqueName: \"kubernetes.io/projected/cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5-kube-api-access-rs92k\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:11:47.373154 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:47.373123 2579 generic.go:356] "Generic (PLEG): container finished" podID="cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5" containerID="686395bfb451da8345b0f20de213d06db6f065515b7346eb03a99e5d99c24647" exitCode=0 Sep 12 02:11:47.373329 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:47.373172 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-98f68e/distinct-2-w4-98f68e-kserve-c5758bc66-d9c6h" Sep 12 02:11:47.373329 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:47.373210 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-98f68e/distinct-2-w4-98f68e-kserve-c5758bc66-d9c6h" event={"ID":"cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5","Type":"ContainerDied","Data":"686395bfb451da8345b0f20de213d06db6f065515b7346eb03a99e5d99c24647"} Sep 12 02:11:47.373329 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:47.373253 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-98f68e/distinct-2-w4-98f68e-kserve-c5758bc66-d9c6h" event={"ID":"cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5","Type":"ContainerDied","Data":"ddbac34853a815eccb65e66f9f428dd9d85513b2e9bdf2e0a124d9d4ff0153c2"} Sep 12 02:11:47.373329 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:47.373270 2579 scope.go:117] "RemoveContainer" containerID="686395bfb451da8345b0f20de213d06db6f065515b7346eb03a99e5d99c24647" Sep 12 02:11:47.385208 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:47.385184 2579 scope.go:117] "RemoveContainer" containerID="686395bfb451da8345b0f20de213d06db6f065515b7346eb03a99e5d99c24647" Sep 12 02:11:47.385629 ip-10-0-133-79 kubenswrapper[2579]: E0912 02:11:47.385601 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"686395bfb451da8345b0f20de213d06db6f065515b7346eb03a99e5d99c24647\": container with ID starting with 686395bfb451da8345b0f20de213d06db6f065515b7346eb03a99e5d99c24647 not found: ID does not exist" containerID="686395bfb451da8345b0f20de213d06db6f065515b7346eb03a99e5d99c24647" Sep 12 02:11:47.385720 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:47.385642 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"686395bfb451da8345b0f20de213d06db6f065515b7346eb03a99e5d99c24647"} err="failed to get container status \"686395bfb451da8345b0f20de213d06db6f065515b7346eb03a99e5d99c24647\": rpc error: code = NotFound desc = could not find container \"686395bfb451da8345b0f20de213d06db6f065515b7346eb03a99e5d99c24647\": container with ID starting with 686395bfb451da8345b0f20de213d06db6f065515b7346eb03a99e5d99c24647 not found: ID does not exist" Sep 12 02:11:47.395323 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:47.395293 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-98f68e/distinct-2-w4-98f68e-kserve-c5758bc66-d9c6h"] Sep 12 02:11:47.397176 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:47.397156 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-98f68e/distinct-2-w4-98f68e-kserve-c5758bc66-d9c6h"] Sep 12 02:11:47.451204 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:47.451170 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ac814d-5d9c46dcff-p9x9x"] Sep 12 02:11:47.451492 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:47.451439 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ac814d-5d9c46dcff-p9x9x" podUID="353e00cb-a98d-49eb-b501-ddd6db06d387" containerName="maas-api" containerID="cri-o://580374478daea9b43b512cd7a5d61c20e1b15fa008dd57669cf12b0a4e591e8b" gracePeriod=30 Sep 12 02:11:47.480858 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:47.480823 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:47.505143 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:47.505113 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ac814d-8d84cc845-8rpfr"] Sep 12 02:11:47.505352 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:47.505308 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ac814d-8d84cc845-8rpfr" podUID="5f96df74-d12b-488f-8eb6-b8445e716021" containerName="payload-processing" containerID="cri-o://a33528f7cf019f3ae7f2efd6c981f0e0ccaff65308deeb16a3a084f04e61b96e" gracePeriod=30 Sep 12 02:11:47.514371 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:47.514332 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ac814d-6c75796f9d-7vsxs"] Sep 12 02:11:47.515171 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:47.514828 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ac814d-6c75796f9d-7vsxs" podUID="2513857c-b802-4096-9159-142f7e339886" containerName="payload-pre-processing" containerID="cri-o://8bd61707dd1744439d462f89ae5a6b5ff97c1abf6eb046fab547f746b71e2fc4" gracePeriod=30 Sep 12 02:11:47.552628 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:47.552583 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:47.616923 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:47.616890 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:47.696967 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:47.696867 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:47.728603 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:47.728575 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5" path="/var/lib/kubelet/pods/cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5/volumes" Sep 12 02:11:47.808667 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:47.808641 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ac814d-5d9c46dcff-p9x9x" Sep 12 02:11:47.835547 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:47.835520 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ac814d-8d84cc845-8rpfr" Sep 12 02:11:47.839274 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:47.839244 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ac814d-6c75796f9d-7vsxs" Sep 12 02:11:47.865837 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:47.865809 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/353e00cb-a98d-49eb-b501-ddd6db06d387-openshift-service-ca\") pod \"353e00cb-a98d-49eb-b501-ddd6db06d387\" (UID: \"353e00cb-a98d-49eb-b501-ddd6db06d387\") " Sep 12 02:11:47.865997 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:47.865884 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2513857c-b802-4096-9159-142f7e339886-plugins-config-volume\") pod \"2513857c-b802-4096-9159-142f7e339886\" (UID: \"2513857c-b802-4096-9159-142f7e339886\") " Sep 12 02:11:47.865997 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:47.865904 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8l62q\" (UniqueName: \"kubernetes.io/projected/353e00cb-a98d-49eb-b501-ddd6db06d387-kube-api-access-8l62q\") pod \"353e00cb-a98d-49eb-b501-ddd6db06d387\" (UID: \"353e00cb-a98d-49eb-b501-ddd6db06d387\") " Sep 12 02:11:47.865997 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:47.865968 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f96df74-d12b-488f-8eb6-b8445e716021-plugins-config-volume\") pod \"5f96df74-d12b-488f-8eb6-b8445e716021\" (UID: \"5f96df74-d12b-488f-8eb6-b8445e716021\") " Sep 12 02:11:47.866134 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:47.866002 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/353e00cb-a98d-49eb-b501-ddd6db06d387-metrics-certs\") pod \"353e00cb-a98d-49eb-b501-ddd6db06d387\" (UID: \"353e00cb-a98d-49eb-b501-ddd6db06d387\") " Sep 12 02:11:47.866134 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:47.866046 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x279s\" (UniqueName: \"kubernetes.io/projected/5f96df74-d12b-488f-8eb6-b8445e716021-kube-api-access-x279s\") pod \"5f96df74-d12b-488f-8eb6-b8445e716021\" (UID: \"5f96df74-d12b-488f-8eb6-b8445e716021\") " Sep 12 02:11:47.866134 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:47.866078 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/353e00cb-a98d-49eb-b501-ddd6db06d387-maas-api-tls\") pod \"353e00cb-a98d-49eb-b501-ddd6db06d387\" (UID: \"353e00cb-a98d-49eb-b501-ddd6db06d387\") " Sep 12 02:11:47.866134 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:47.866126 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2rsmb\" (UniqueName: \"kubernetes.io/projected/2513857c-b802-4096-9159-142f7e339886-kube-api-access-2rsmb\") pod \"2513857c-b802-4096-9159-142f7e339886\" (UID: \"2513857c-b802-4096-9159-142f7e339886\") " Sep 12 02:11:47.866334 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:47.866161 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/353e00cb-a98d-49eb-b501-ddd6db06d387-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "353e00cb-a98d-49eb-b501-ddd6db06d387" (UID: "353e00cb-a98d-49eb-b501-ddd6db06d387"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 12 02:11:47.866402 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:47.866350 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f96df74-d12b-488f-8eb6-b8445e716021-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5f96df74-d12b-488f-8eb6-b8445e716021" (UID: "5f96df74-d12b-488f-8eb6-b8445e716021"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 12 02:11:47.866543 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:47.866524 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f96df74-d12b-488f-8eb6-b8445e716021-plugins-config-volume\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:11:47.866625 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:47.866548 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/353e00cb-a98d-49eb-b501-ddd6db06d387-openshift-service-ca\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:11:47.866625 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:47.866577 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2513857c-b802-4096-9159-142f7e339886-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2513857c-b802-4096-9159-142f7e339886" (UID: "2513857c-b802-4096-9159-142f7e339886"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 12 02:11:47.868518 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:47.868425 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/353e00cb-a98d-49eb-b501-ddd6db06d387-kube-api-access-8l62q" (OuterVolumeSpecName: "kube-api-access-8l62q") pod "353e00cb-a98d-49eb-b501-ddd6db06d387" (UID: "353e00cb-a98d-49eb-b501-ddd6db06d387"). InnerVolumeSpecName "kube-api-access-8l62q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 12 02:11:47.870443 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:47.869317 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/353e00cb-a98d-49eb-b501-ddd6db06d387-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "353e00cb-a98d-49eb-b501-ddd6db06d387" (UID: "353e00cb-a98d-49eb-b501-ddd6db06d387"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 12 02:11:47.870443 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:47.869786 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f96df74-d12b-488f-8eb6-b8445e716021-kube-api-access-x279s" (OuterVolumeSpecName: "kube-api-access-x279s") pod "5f96df74-d12b-488f-8eb6-b8445e716021" (UID: "5f96df74-d12b-488f-8eb6-b8445e716021"). InnerVolumeSpecName "kube-api-access-x279s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 12 02:11:47.870443 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:47.870133 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2513857c-b802-4096-9159-142f7e339886-kube-api-access-2rsmb" (OuterVolumeSpecName: "kube-api-access-2rsmb") pod "2513857c-b802-4096-9159-142f7e339886" (UID: "2513857c-b802-4096-9159-142f7e339886"). InnerVolumeSpecName "kube-api-access-2rsmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 12 02:11:47.870855 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:47.870827 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/353e00cb-a98d-49eb-b501-ddd6db06d387-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "353e00cb-a98d-49eb-b501-ddd6db06d387" (UID: "353e00cb-a98d-49eb-b501-ddd6db06d387"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 12 02:11:47.968014 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:47.967937 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l62q\" (UniqueName: \"kubernetes.io/projected/353e00cb-a98d-49eb-b501-ddd6db06d387-kube-api-access-8l62q\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:11:47.968014 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:47.967962 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/353e00cb-a98d-49eb-b501-ddd6db06d387-metrics-certs\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:11:47.968014 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:47.967972 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x279s\" (UniqueName: \"kubernetes.io/projected/5f96df74-d12b-488f-8eb6-b8445e716021-kube-api-access-x279s\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:11:47.968014 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:47.967980 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/353e00cb-a98d-49eb-b501-ddd6db06d387-maas-api-tls\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:11:47.968014 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:47.967989 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rsmb\" (UniqueName: \"kubernetes.io/projected/2513857c-b802-4096-9159-142f7e339886-kube-api-access-2rsmb\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:11:47.968014 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:47.967996 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2513857c-b802-4096-9159-142f7e339886-plugins-config-volume\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:11:48.049303 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:48.049252 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:48.149631 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:48.149597 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:48.380265 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:48.380234 2579 generic.go:356] "Generic (PLEG): container finished" podID="2513857c-b802-4096-9159-142f7e339886" containerID="8bd61707dd1744439d462f89ae5a6b5ff97c1abf6eb046fab547f746b71e2fc4" exitCode=0 Sep 12 02:11:48.380474 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:48.380271 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ac814d-6c75796f9d-7vsxs" Sep 12 02:11:48.380474 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:48.380325 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ac814d-6c75796f9d-7vsxs" event={"ID":"2513857c-b802-4096-9159-142f7e339886","Type":"ContainerDied","Data":"8bd61707dd1744439d462f89ae5a6b5ff97c1abf6eb046fab547f746b71e2fc4"} Sep 12 02:11:48.380474 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:48.380379 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ac814d-6c75796f9d-7vsxs" event={"ID":"2513857c-b802-4096-9159-142f7e339886","Type":"ContainerDied","Data":"42a32aaa8bdb5f775c8fe800be91ef130efe58e2248c1ae79a98455129f2c61e"} Sep 12 02:11:48.380474 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:48.380406 2579 scope.go:117] "RemoveContainer" containerID="8bd61707dd1744439d462f89ae5a6b5ff97c1abf6eb046fab547f746b71e2fc4" Sep 12 02:11:48.381728 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:48.381703 2579 generic.go:356] "Generic (PLEG): container finished" podID="353e00cb-a98d-49eb-b501-ddd6db06d387" containerID="580374478daea9b43b512cd7a5d61c20e1b15fa008dd57669cf12b0a4e591e8b" exitCode=0 Sep 12 02:11:48.381835 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:48.381753 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ac814d-5d9c46dcff-p9x9x" Sep 12 02:11:48.381835 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:48.381806 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ac814d-5d9c46dcff-p9x9x" event={"ID":"353e00cb-a98d-49eb-b501-ddd6db06d387","Type":"ContainerDied","Data":"580374478daea9b43b512cd7a5d61c20e1b15fa008dd57669cf12b0a4e591e8b"} Sep 12 02:11:48.381947 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:48.381860 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ac814d-5d9c46dcff-p9x9x" event={"ID":"353e00cb-a98d-49eb-b501-ddd6db06d387","Type":"ContainerDied","Data":"31bb878eb3b14a55a7a9932cdced8dce730d8caf837ee36f397b9b52ecb99106"} Sep 12 02:11:48.384308 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:48.384282 2579 generic.go:356] "Generic (PLEG): container finished" podID="5f96df74-d12b-488f-8eb6-b8445e716021" containerID="a33528f7cf019f3ae7f2efd6c981f0e0ccaff65308deeb16a3a084f04e61b96e" exitCode=0 Sep 12 02:11:48.385233 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:48.384325 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ac814d-8d84cc845-8rpfr" Sep 12 02:11:48.385233 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:48.384335 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ac814d-8d84cc845-8rpfr" event={"ID":"5f96df74-d12b-488f-8eb6-b8445e716021","Type":"ContainerDied","Data":"a33528f7cf019f3ae7f2efd6c981f0e0ccaff65308deeb16a3a084f04e61b96e"} Sep 12 02:11:48.385233 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:48.384371 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ac814d-8d84cc845-8rpfr" event={"ID":"5f96df74-d12b-488f-8eb6-b8445e716021","Type":"ContainerDied","Data":"dc608dbbdbc3363b7e0336cc0f5506b12b31168f9d339c33786c01480ec069a3"} Sep 12 02:11:48.392063 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:48.392042 2579 scope.go:117] "RemoveContainer" containerID="8bd61707dd1744439d462f89ae5a6b5ff97c1abf6eb046fab547f746b71e2fc4" Sep 12 02:11:48.392374 ip-10-0-133-79 kubenswrapper[2579]: E0912 02:11:48.392357 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bd61707dd1744439d462f89ae5a6b5ff97c1abf6eb046fab547f746b71e2fc4\": container with ID starting with 8bd61707dd1744439d462f89ae5a6b5ff97c1abf6eb046fab547f746b71e2fc4 not found: ID does not exist" containerID="8bd61707dd1744439d462f89ae5a6b5ff97c1abf6eb046fab547f746b71e2fc4" Sep 12 02:11:48.392471 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:48.392383 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bd61707dd1744439d462f89ae5a6b5ff97c1abf6eb046fab547f746b71e2fc4"} err="failed to get container status \"8bd61707dd1744439d462f89ae5a6b5ff97c1abf6eb046fab547f746b71e2fc4\": rpc error: code = NotFound desc = could not find container \"8bd61707dd1744439d462f89ae5a6b5ff97c1abf6eb046fab547f746b71e2fc4\": container with ID starting with 8bd61707dd1744439d462f89ae5a6b5ff97c1abf6eb046fab547f746b71e2fc4 not found: ID does not exist" Sep 12 02:11:48.392471 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:48.392398 2579 scope.go:117] "RemoveContainer" containerID="580374478daea9b43b512cd7a5d61c20e1b15fa008dd57669cf12b0a4e591e8b" Sep 12 02:11:48.401429 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:48.401412 2579 scope.go:117] "RemoveContainer" containerID="580374478daea9b43b512cd7a5d61c20e1b15fa008dd57669cf12b0a4e591e8b" Sep 12 02:11:48.401741 ip-10-0-133-79 kubenswrapper[2579]: E0912 02:11:48.401723 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"580374478daea9b43b512cd7a5d61c20e1b15fa008dd57669cf12b0a4e591e8b\": container with ID starting with 580374478daea9b43b512cd7a5d61c20e1b15fa008dd57669cf12b0a4e591e8b not found: ID does not exist" containerID="580374478daea9b43b512cd7a5d61c20e1b15fa008dd57669cf12b0a4e591e8b" Sep 12 02:11:48.401790 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:48.401749 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"580374478daea9b43b512cd7a5d61c20e1b15fa008dd57669cf12b0a4e591e8b"} err="failed to get container status \"580374478daea9b43b512cd7a5d61c20e1b15fa008dd57669cf12b0a4e591e8b\": rpc error: code = NotFound desc = could not find container \"580374478daea9b43b512cd7a5d61c20e1b15fa008dd57669cf12b0a4e591e8b\": container with ID starting with 580374478daea9b43b512cd7a5d61c20e1b15fa008dd57669cf12b0a4e591e8b not found: ID does not exist" Sep 12 02:11:48.401790 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:48.401766 2579 scope.go:117] "RemoveContainer" containerID="a33528f7cf019f3ae7f2efd6c981f0e0ccaff65308deeb16a3a084f04e61b96e" Sep 12 02:11:48.410040 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:48.410015 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ac814d-6c75796f9d-7vsxs"] Sep 12 02:11:48.411145 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:48.411124 2579 scope.go:117] "RemoveContainer" containerID="a33528f7cf019f3ae7f2efd6c981f0e0ccaff65308deeb16a3a084f04e61b96e" Sep 12 02:11:48.411632 ip-10-0-133-79 kubenswrapper[2579]: E0912 02:11:48.411606 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a33528f7cf019f3ae7f2efd6c981f0e0ccaff65308deeb16a3a084f04e61b96e\": container with ID starting with a33528f7cf019f3ae7f2efd6c981f0e0ccaff65308deeb16a3a084f04e61b96e not found: ID does not exist" containerID="a33528f7cf019f3ae7f2efd6c981f0e0ccaff65308deeb16a3a084f04e61b96e" Sep 12 02:11:48.411739 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:48.411689 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a33528f7cf019f3ae7f2efd6c981f0e0ccaff65308deeb16a3a084f04e61b96e"} err="failed to get container status \"a33528f7cf019f3ae7f2efd6c981f0e0ccaff65308deeb16a3a084f04e61b96e\": rpc error: code = NotFound desc = could not find container \"a33528f7cf019f3ae7f2efd6c981f0e0ccaff65308deeb16a3a084f04e61b96e\": container with ID starting with a33528f7cf019f3ae7f2efd6c981f0e0ccaff65308deeb16a3a084f04e61b96e not found: ID does not exist" Sep 12 02:11:48.413867 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:48.413847 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ac814d-6c75796f9d-7vsxs"] Sep 12 02:11:48.424580 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:48.424558 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ac814d-8d84cc845-8rpfr"] Sep 12 02:11:48.428224 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:48.428201 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ac814d-8d84cc845-8rpfr"] Sep 12 02:11:48.438770 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:48.438747 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ac814d-5d9c46dcff-p9x9x"] Sep 12 02:11:48.443114 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:48.443091 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ac814d-5d9c46dcff-p9x9x"] Sep 12 02:11:48.752821 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:48.752791 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:49.469470 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:49.469426 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:49.726739 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:49.726663 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2513857c-b802-4096-9159-142f7e339886" path="/var/lib/kubelet/pods/2513857c-b802-4096-9159-142f7e339886/volumes" Sep 12 02:11:49.727086 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:49.727072 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="353e00cb-a98d-49eb-b501-ddd6db06d387" path="/var/lib/kubelet/pods/353e00cb-a98d-49eb-b501-ddd6db06d387/volumes" Sep 12 02:11:49.727385 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:49.727373 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f96df74-d12b-488f-8eb6-b8445e716021" path="/var/lib/kubelet/pods/5f96df74-d12b-488f-8eb6-b8445e716021/volumes" Sep 12 02:11:50.465233 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:50.465191 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:50.565832 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:50.565793 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:50.779385 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:50.779299 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:53.841337 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:53.841294 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:53.940468 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:53.940425 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:11:54.113085 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:11:54.112988 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:12:05.024925 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:12:05.024889 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:12:08.481231 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:12:08.481189 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:12:08.582577 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:12:08.582542 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:12:09.660712 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:12:09.660682 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:12:15.216649 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:12:15.216618 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:12:24.279404 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:12:24.279358 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:12:25.699667 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:12:25.699630 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-fbb6d9/auth-test-model-fbb6d9-kserve-577d59df55-pk7g9"] Sep 12 02:12:25.723915 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:12:25.723884 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:12:28.422206 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:12:28.422171 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:12:31.551717 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:12:31.551679 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:12:33.239174 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:12:33.239144 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:12:35.130262 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:12:35.130224 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:12:40.720914 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:12:40.720875 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-fbb6d9/auth-test-model-fbb6d9-kserve-577d59df55-pk7g9" podUID="8fe29199-dd9d-4326-8a67-d1ca0e11544f" containerName="main" containerID="cri-o://d98c0f71c40a698b9c7576d771f0efb0374349efe4cb4f2f863b43f700bdcc2e" gracePeriod=45 Sep 12 02:12:40.974911 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:12:40.974854 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-fbb6d9/auth-test-model-fbb6d9-kserve-577d59df55-pk7g9" Sep 12 02:12:41.099539 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:12:41.099504 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8ffqz\" (UniqueName: \"kubernetes.io/projected/8fe29199-dd9d-4326-8a67-d1ca0e11544f-kube-api-access-8ffqz\") pod \"8fe29199-dd9d-4326-8a67-d1ca0e11544f\" (UID: \"8fe29199-dd9d-4326-8a67-d1ca0e11544f\") " Sep 12 02:12:41.099708 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:12:41.099579 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fe29199-dd9d-4326-8a67-d1ca0e11544f-model-cache\") pod \"8fe29199-dd9d-4326-8a67-d1ca0e11544f\" (UID: \"8fe29199-dd9d-4326-8a67-d1ca0e11544f\") " Sep 12 02:12:41.099708 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:12:41.099670 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fe29199-dd9d-4326-8a67-d1ca0e11544f-tls-certs\") pod \"8fe29199-dd9d-4326-8a67-d1ca0e11544f\" (UID: \"8fe29199-dd9d-4326-8a67-d1ca0e11544f\") " Sep 12 02:12:41.099708 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:12:41.099701 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fe29199-dd9d-4326-8a67-d1ca0e11544f-tmp-dir\") pod \"8fe29199-dd9d-4326-8a67-d1ca0e11544f\" (UID: \"8fe29199-dd9d-4326-8a67-d1ca0e11544f\") " Sep 12 02:12:41.099864 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:12:41.099743 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fe29199-dd9d-4326-8a67-d1ca0e11544f-dshm\") pod \"8fe29199-dd9d-4326-8a67-d1ca0e11544f\" (UID: \"8fe29199-dd9d-4326-8a67-d1ca0e11544f\") " Sep 12 02:12:41.099864 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:12:41.099791 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fe29199-dd9d-4326-8a67-d1ca0e11544f-home\") pod \"8fe29199-dd9d-4326-8a67-d1ca0e11544f\" (UID: \"8fe29199-dd9d-4326-8a67-d1ca0e11544f\") " Sep 12 02:12:41.099864 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:12:41.099800 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fe29199-dd9d-4326-8a67-d1ca0e11544f-model-cache" (OuterVolumeSpecName: "model-cache") pod "8fe29199-dd9d-4326-8a67-d1ca0e11544f" (UID: "8fe29199-dd9d-4326-8a67-d1ca0e11544f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 12 02:12:41.100021 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:12:41.099978 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fe29199-dd9d-4326-8a67-d1ca0e11544f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8fe29199-dd9d-4326-8a67-d1ca0e11544f" (UID: "8fe29199-dd9d-4326-8a67-d1ca0e11544f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 12 02:12:41.100095 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:12:41.100027 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fe29199-dd9d-4326-8a67-d1ca0e11544f-home" (OuterVolumeSpecName: "home") pod "8fe29199-dd9d-4326-8a67-d1ca0e11544f" (UID: "8fe29199-dd9d-4326-8a67-d1ca0e11544f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 12 02:12:41.100220 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:12:41.100204 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fe29199-dd9d-4326-8a67-d1ca0e11544f-home\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:12:41.100485 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:12:41.100223 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fe29199-dd9d-4326-8a67-d1ca0e11544f-model-cache\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:12:41.100485 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:12:41.100234 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fe29199-dd9d-4326-8a67-d1ca0e11544f-tmp-dir\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:12:41.101935 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:12:41.101905 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fe29199-dd9d-4326-8a67-d1ca0e11544f-kube-api-access-8ffqz" (OuterVolumeSpecName: "kube-api-access-8ffqz") pod "8fe29199-dd9d-4326-8a67-d1ca0e11544f" (UID: "8fe29199-dd9d-4326-8a67-d1ca0e11544f"). InnerVolumeSpecName "kube-api-access-8ffqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 12 02:12:41.101935 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:12:41.101921 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fe29199-dd9d-4326-8a67-d1ca0e11544f-dshm" (OuterVolumeSpecName: "dshm") pod "8fe29199-dd9d-4326-8a67-d1ca0e11544f" (UID: "8fe29199-dd9d-4326-8a67-d1ca0e11544f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 12 02:12:41.102088 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:12:41.101955 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fe29199-dd9d-4326-8a67-d1ca0e11544f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8fe29199-dd9d-4326-8a67-d1ca0e11544f" (UID: "8fe29199-dd9d-4326-8a67-d1ca0e11544f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 12 02:12:41.201336 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:12:41.201289 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fe29199-dd9d-4326-8a67-d1ca0e11544f-tls-certs\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:12:41.201336 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:12:41.201327 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fe29199-dd9d-4326-8a67-d1ca0e11544f-dshm\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:12:41.201336 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:12:41.201339 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ffqz\" (UniqueName: \"kubernetes.io/projected/8fe29199-dd9d-4326-8a67-d1ca0e11544f-kube-api-access-8ffqz\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:12:41.671986 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:12:41.671948 2579 generic.go:356] "Generic (PLEG): container finished" podID="8fe29199-dd9d-4326-8a67-d1ca0e11544f" containerID="d98c0f71c40a698b9c7576d771f0efb0374349efe4cb4f2f863b43f700bdcc2e" exitCode=0 Sep 12 02:12:41.671986 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:12:41.671988 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-fbb6d9/auth-test-model-fbb6d9-kserve-577d59df55-pk7g9" Sep 12 02:12:41.672204 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:12:41.672032 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-fbb6d9/auth-test-model-fbb6d9-kserve-577d59df55-pk7g9" event={"ID":"8fe29199-dd9d-4326-8a67-d1ca0e11544f","Type":"ContainerDied","Data":"d98c0f71c40a698b9c7576d771f0efb0374349efe4cb4f2f863b43f700bdcc2e"} Sep 12 02:12:41.672204 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:12:41.672076 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-fbb6d9/auth-test-model-fbb6d9-kserve-577d59df55-pk7g9" event={"ID":"8fe29199-dd9d-4326-8a67-d1ca0e11544f","Type":"ContainerDied","Data":"82933601679beb1eb76f0c3c3c9ca49349f523aa9621492627643de14c3d1668"} Sep 12 02:12:41.672204 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:12:41.672094 2579 scope.go:117] "RemoveContainer" containerID="d98c0f71c40a698b9c7576d771f0efb0374349efe4cb4f2f863b43f700bdcc2e" Sep 12 02:12:41.681414 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:12:41.681399 2579 scope.go:117] "RemoveContainer" containerID="d98c0f71c40a698b9c7576d771f0efb0374349efe4cb4f2f863b43f700bdcc2e" Sep 12 02:12:41.681718 ip-10-0-133-79 kubenswrapper[2579]: E0912 02:12:41.681700 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d98c0f71c40a698b9c7576d771f0efb0374349efe4cb4f2f863b43f700bdcc2e\": container with ID starting with d98c0f71c40a698b9c7576d771f0efb0374349efe4cb4f2f863b43f700bdcc2e not found: ID does not exist" containerID="d98c0f71c40a698b9c7576d771f0efb0374349efe4cb4f2f863b43f700bdcc2e" Sep 12 02:12:41.681793 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:12:41.681727 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d98c0f71c40a698b9c7576d771f0efb0374349efe4cb4f2f863b43f700bdcc2e"} err="failed to get container status \"d98c0f71c40a698b9c7576d771f0efb0374349efe4cb4f2f863b43f700bdcc2e\": rpc error: code = NotFound desc = could not find container \"d98c0f71c40a698b9c7576d771f0efb0374349efe4cb4f2f863b43f700bdcc2e\": container with ID starting with d98c0f71c40a698b9c7576d771f0efb0374349efe4cb4f2f863b43f700bdcc2e not found: ID does not exist" Sep 12 02:12:41.692585 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:12:41.692562 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-fbb6d9/auth-test-model-fbb6d9-kserve-577d59df55-pk7g9"] Sep 12 02:12:41.696050 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:12:41.696029 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-fbb6d9/auth-test-model-fbb6d9-kserve-577d59df55-pk7g9"] Sep 12 02:12:41.726292 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:12:41.726267 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fe29199-dd9d-4326-8a67-d1ca0e11544f" path="/var/lib/kubelet/pods/8fe29199-dd9d-4326-8a67-d1ca0e11544f/volumes" Sep 12 02:13:02.426153 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.426120 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-fbb6d9/sub-test-model-fbb6d9-kserve-7548855b8f-k7dnz"] Sep 12 02:13:02.426759 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.426646 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c105641-487e-4a89-a4d5-a4c2b9206411" containerName="maas-api" Sep 12 02:13:02.426759 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.426662 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c105641-487e-4a89-a4d5-a4c2b9206411" containerName="maas-api" Sep 12 02:13:02.426759 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.426671 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f1890ff-65c0-4771-9d6b-6887073fa37f" containerName="payload-pre-processing" Sep 12 02:13:02.426759 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.426677 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f1890ff-65c0-4771-9d6b-6887073fa37f" containerName="payload-pre-processing" Sep 12 02:13:02.426759 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.426686 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c273cea1-7278-4513-8432-ccba4c25c999" containerName="payload-pre-processing" Sep 12 02:13:02.426759 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.426691 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c273cea1-7278-4513-8432-ccba4c25c999" containerName="payload-pre-processing" Sep 12 02:13:02.426759 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.426696 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c9f8ad4-58c8-45a9-8183-00a5976e1f54" containerName="maas-api" Sep 12 02:13:02.426759 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.426701 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c9f8ad4-58c8-45a9-8183-00a5976e1f54" containerName="maas-api" Sep 12 02:13:02.426759 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.426716 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5" containerName="main" Sep 12 02:13:02.426759 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.426722 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5" containerName="main" Sep 12 02:13:02.426759 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.426728 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a007efd1-b179-445a-84ea-28cf8f744c9b" containerName="payload-pre-processing" Sep 12 02:13:02.426759 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.426733 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a007efd1-b179-445a-84ea-28cf8f744c9b" containerName="payload-pre-processing" Sep 12 02:13:02.426759 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.426739 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2513857c-b802-4096-9159-142f7e339886" containerName="payload-pre-processing" Sep 12 02:13:02.426759 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.426746 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2513857c-b802-4096-9159-142f7e339886" containerName="payload-pre-processing" Sep 12 02:13:02.426759 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.426755 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f96df74-d12b-488f-8eb6-b8445e716021" containerName="payload-processing" Sep 12 02:13:02.426759 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.426760 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f96df74-d12b-488f-8eb6-b8445e716021" containerName="payload-processing" Sep 12 02:13:02.427251 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.426769 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18093353-67b9-4a8b-9b50-b07ff688f925" containerName="main" Sep 12 02:13:02.427251 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.426774 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="18093353-67b9-4a8b-9b50-b07ff688f925" containerName="main" Sep 12 02:13:02.427251 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.426780 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="353e00cb-a98d-49eb-b501-ddd6db06d387" containerName="maas-api" Sep 12 02:13:02.427251 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.426784 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="353e00cb-a98d-49eb-b501-ddd6db06d387" containerName="maas-api" Sep 12 02:13:02.427251 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.426793 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c25279e-6e3a-4104-9934-a6707fa63885" containerName="main" Sep 12 02:13:02.427251 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.426797 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c25279e-6e3a-4104-9934-a6707fa63885" containerName="main" Sep 12 02:13:02.427251 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.426803 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="132fc085-122b-48ab-b07e-dd81c6c1c516" containerName="payload-pre-processing" Sep 12 02:13:02.427251 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.426807 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="132fc085-122b-48ab-b07e-dd81c6c1c516" containerName="payload-pre-processing" Sep 12 02:13:02.427251 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.426814 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fe29199-dd9d-4326-8a67-d1ca0e11544f" containerName="main" Sep 12 02:13:02.427251 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.426818 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fe29199-dd9d-4326-8a67-d1ca0e11544f" containerName="main" Sep 12 02:13:02.427251 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.426873 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c273cea1-7278-4513-8432-ccba4c25c999" containerName="payload-pre-processing" Sep 12 02:13:02.427251 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.426882 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="18093353-67b9-4a8b-9b50-b07ff688f925" containerName="main" Sep 12 02:13:02.427251 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.426887 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c105641-487e-4a89-a4d5-a4c2b9206411" containerName="maas-api" Sep 12 02:13:02.427251 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.426894 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c25279e-6e3a-4104-9934-a6707fa63885" containerName="main" Sep 12 02:13:02.427251 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.426899 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2513857c-b802-4096-9159-142f7e339886" containerName="payload-pre-processing" Sep 12 02:13:02.427251 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.426904 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc84d075-65c5-4dcf-9f6f-6cd0737ef9d5" containerName="main" Sep 12 02:13:02.427251 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.426910 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f96df74-d12b-488f-8eb6-b8445e716021" containerName="payload-processing" Sep 12 02:13:02.427251 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.426915 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="353e00cb-a98d-49eb-b501-ddd6db06d387" containerName="maas-api" Sep 12 02:13:02.427251 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.426920 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fe29199-dd9d-4326-8a67-d1ca0e11544f" containerName="main" Sep 12 02:13:02.427251 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.426925 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a007efd1-b179-445a-84ea-28cf8f744c9b" containerName="payload-pre-processing" Sep 12 02:13:02.427251 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.426932 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f1890ff-65c0-4771-9d6b-6887073fa37f" containerName="payload-pre-processing" Sep 12 02:13:02.427251 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.426938 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="132fc085-122b-48ab-b07e-dd81c6c1c516" containerName="payload-pre-processing" Sep 12 02:13:02.427251 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.426943 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c9f8ad4-58c8-45a9-8183-00a5976e1f54" containerName="maas-api" Sep 12 02:13:02.431227 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.431203 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-fbb6d9/sub-test-model-fbb6d9-kserve-7548855b8f-k7dnz" Sep 12 02:13:02.434570 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.434534 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-fbb6d9\"/\"sub-test-model-fbb6d9-kserve-self-signed-certs\"" Sep 12 02:13:02.434570 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.434549 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-fbb6d9\"/\"kube-root-ca.crt\"" Sep 12 02:13:02.434766 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.434549 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-fbb6d9\"/\"default-dockercfg-mjwfn\"" Sep 12 02:13:02.434766 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.434549 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-fbb6d9\"/\"openshift-service-ca.crt\"" Sep 12 02:13:02.438357 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.438336 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-fbb6d9/sub-test-model-fbb6d9-kserve-7548855b8f-k7dnz"] Sep 12 02:13:02.455755 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.455707 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:13:02.507862 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.507827 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/545773bc-bdc3-4b7d-8a44-af57a9c5a368-model-cache\") pod \"sub-test-model-fbb6d9-kserve-7548855b8f-k7dnz\" (UID: \"545773bc-bdc3-4b7d-8a44-af57a9c5a368\") " pod="ai-tenant-e2e-shared-b-w3-fbb6d9/sub-test-model-fbb6d9-kserve-7548855b8f-k7dnz" Sep 12 02:13:02.508022 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.507865 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/545773bc-bdc3-4b7d-8a44-af57a9c5a368-tmp-dir\") pod \"sub-test-model-fbb6d9-kserve-7548855b8f-k7dnz\" (UID: \"545773bc-bdc3-4b7d-8a44-af57a9c5a368\") " pod="ai-tenant-e2e-shared-b-w3-fbb6d9/sub-test-model-fbb6d9-kserve-7548855b8f-k7dnz" Sep 12 02:13:02.508022 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.507974 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/545773bc-bdc3-4b7d-8a44-af57a9c5a368-dshm\") pod \"sub-test-model-fbb6d9-kserve-7548855b8f-k7dnz\" (UID: \"545773bc-bdc3-4b7d-8a44-af57a9c5a368\") " pod="ai-tenant-e2e-shared-b-w3-fbb6d9/sub-test-model-fbb6d9-kserve-7548855b8f-k7dnz" Sep 12 02:13:02.508094 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.508030 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/545773bc-bdc3-4b7d-8a44-af57a9c5a368-home\") pod \"sub-test-model-fbb6d9-kserve-7548855b8f-k7dnz\" (UID: \"545773bc-bdc3-4b7d-8a44-af57a9c5a368\") " pod="ai-tenant-e2e-shared-b-w3-fbb6d9/sub-test-model-fbb6d9-kserve-7548855b8f-k7dnz" Sep 12 02:13:02.508094 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.508052 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxdck\" (UniqueName: \"kubernetes.io/projected/545773bc-bdc3-4b7d-8a44-af57a9c5a368-kube-api-access-fxdck\") pod \"sub-test-model-fbb6d9-kserve-7548855b8f-k7dnz\" (UID: \"545773bc-bdc3-4b7d-8a44-af57a9c5a368\") " pod="ai-tenant-e2e-shared-b-w3-fbb6d9/sub-test-model-fbb6d9-kserve-7548855b8f-k7dnz" Sep 12 02:13:02.508094 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.508076 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/545773bc-bdc3-4b7d-8a44-af57a9c5a368-tls-certs\") pod \"sub-test-model-fbb6d9-kserve-7548855b8f-k7dnz\" (UID: \"545773bc-bdc3-4b7d-8a44-af57a9c5a368\") " pod="ai-tenant-e2e-shared-b-w3-fbb6d9/sub-test-model-fbb6d9-kserve-7548855b8f-k7dnz" Sep 12 02:13:02.608874 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.608842 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/545773bc-bdc3-4b7d-8a44-af57a9c5a368-home\") pod \"sub-test-model-fbb6d9-kserve-7548855b8f-k7dnz\" (UID: \"545773bc-bdc3-4b7d-8a44-af57a9c5a368\") " pod="ai-tenant-e2e-shared-b-w3-fbb6d9/sub-test-model-fbb6d9-kserve-7548855b8f-k7dnz" Sep 12 02:13:02.608874 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.608879 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxdck\" (UniqueName: \"kubernetes.io/projected/545773bc-bdc3-4b7d-8a44-af57a9c5a368-kube-api-access-fxdck\") pod \"sub-test-model-fbb6d9-kserve-7548855b8f-k7dnz\" (UID: \"545773bc-bdc3-4b7d-8a44-af57a9c5a368\") " pod="ai-tenant-e2e-shared-b-w3-fbb6d9/sub-test-model-fbb6d9-kserve-7548855b8f-k7dnz" Sep 12 02:13:02.609135 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.608913 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/545773bc-bdc3-4b7d-8a44-af57a9c5a368-tls-certs\") pod \"sub-test-model-fbb6d9-kserve-7548855b8f-k7dnz\" (UID: \"545773bc-bdc3-4b7d-8a44-af57a9c5a368\") " pod="ai-tenant-e2e-shared-b-w3-fbb6d9/sub-test-model-fbb6d9-kserve-7548855b8f-k7dnz" Sep 12 02:13:02.609135 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.608951 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/545773bc-bdc3-4b7d-8a44-af57a9c5a368-model-cache\") pod \"sub-test-model-fbb6d9-kserve-7548855b8f-k7dnz\" (UID: \"545773bc-bdc3-4b7d-8a44-af57a9c5a368\") " pod="ai-tenant-e2e-shared-b-w3-fbb6d9/sub-test-model-fbb6d9-kserve-7548855b8f-k7dnz" Sep 12 02:13:02.609135 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.608973 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/545773bc-bdc3-4b7d-8a44-af57a9c5a368-tmp-dir\") pod \"sub-test-model-fbb6d9-kserve-7548855b8f-k7dnz\" (UID: \"545773bc-bdc3-4b7d-8a44-af57a9c5a368\") " pod="ai-tenant-e2e-shared-b-w3-fbb6d9/sub-test-model-fbb6d9-kserve-7548855b8f-k7dnz" Sep 12 02:13:02.609135 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.609052 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/545773bc-bdc3-4b7d-8a44-af57a9c5a368-dshm\") pod \"sub-test-model-fbb6d9-kserve-7548855b8f-k7dnz\" (UID: \"545773bc-bdc3-4b7d-8a44-af57a9c5a368\") " pod="ai-tenant-e2e-shared-b-w3-fbb6d9/sub-test-model-fbb6d9-kserve-7548855b8f-k7dnz" Sep 12 02:13:02.609313 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.609256 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/545773bc-bdc3-4b7d-8a44-af57a9c5a368-home\") pod \"sub-test-model-fbb6d9-kserve-7548855b8f-k7dnz\" (UID: \"545773bc-bdc3-4b7d-8a44-af57a9c5a368\") " pod="ai-tenant-e2e-shared-b-w3-fbb6d9/sub-test-model-fbb6d9-kserve-7548855b8f-k7dnz" Sep 12 02:13:02.609350 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.609311 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/545773bc-bdc3-4b7d-8a44-af57a9c5a368-model-cache\") pod \"sub-test-model-fbb6d9-kserve-7548855b8f-k7dnz\" (UID: \"545773bc-bdc3-4b7d-8a44-af57a9c5a368\") " pod="ai-tenant-e2e-shared-b-w3-fbb6d9/sub-test-model-fbb6d9-kserve-7548855b8f-k7dnz" Sep 12 02:13:02.609385 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.609357 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/545773bc-bdc3-4b7d-8a44-af57a9c5a368-tmp-dir\") pod \"sub-test-model-fbb6d9-kserve-7548855b8f-k7dnz\" (UID: \"545773bc-bdc3-4b7d-8a44-af57a9c5a368\") " pod="ai-tenant-e2e-shared-b-w3-fbb6d9/sub-test-model-fbb6d9-kserve-7548855b8f-k7dnz" Sep 12 02:13:02.611261 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.611243 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/545773bc-bdc3-4b7d-8a44-af57a9c5a368-dshm\") pod \"sub-test-model-fbb6d9-kserve-7548855b8f-k7dnz\" (UID: \"545773bc-bdc3-4b7d-8a44-af57a9c5a368\") " pod="ai-tenant-e2e-shared-b-w3-fbb6d9/sub-test-model-fbb6d9-kserve-7548855b8f-k7dnz" Sep 12 02:13:02.611681 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.611659 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/545773bc-bdc3-4b7d-8a44-af57a9c5a368-tls-certs\") pod \"sub-test-model-fbb6d9-kserve-7548855b8f-k7dnz\" (UID: \"545773bc-bdc3-4b7d-8a44-af57a9c5a368\") " pod="ai-tenant-e2e-shared-b-w3-fbb6d9/sub-test-model-fbb6d9-kserve-7548855b8f-k7dnz" Sep 12 02:13:02.617072 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.617050 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxdck\" (UniqueName: \"kubernetes.io/projected/545773bc-bdc3-4b7d-8a44-af57a9c5a368-kube-api-access-fxdck\") pod \"sub-test-model-fbb6d9-kserve-7548855b8f-k7dnz\" (UID: \"545773bc-bdc3-4b7d-8a44-af57a9c5a368\") " pod="ai-tenant-e2e-shared-b-w3-fbb6d9/sub-test-model-fbb6d9-kserve-7548855b8f-k7dnz" Sep 12 02:13:02.743286 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:02.743197 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-fbb6d9/sub-test-model-fbb6d9-kserve-7548855b8f-k7dnz" Sep 12 02:13:03.014747 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:03.014705 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-fbb6d9/sub-test-model-fbb6d9-kserve-7548855b8f-k7dnz"] Sep 12 02:13:03.017170 ip-10-0-133-79 kubenswrapper[2579]: W0912 02:13:03.017134 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod545773bc_bdc3_4b7d_8a44_af57a9c5a368.slice/crio-35eaf1c5fb5d30e835befb8a51e2ae3a599d06d02b4a63de957f443e918c6dad WatchSource:0}: Error finding container 35eaf1c5fb5d30e835befb8a51e2ae3a599d06d02b4a63de957f443e918c6dad: Status 404 returned error can't find the container with id 35eaf1c5fb5d30e835befb8a51e2ae3a599d06d02b4a63de957f443e918c6dad Sep 12 02:13:03.784809 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:03.784769 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-fbb6d9/sub-test-model-fbb6d9-kserve-7548855b8f-k7dnz" event={"ID":"545773bc-bdc3-4b7d-8a44-af57a9c5a368","Type":"ContainerStarted","Data":"025066ed600a1b44851afeeb4dbc2c4b75f1fa41e50aba1f60219101a8f9ebbd"} Sep 12 02:13:03.784809 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:03.784814 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-fbb6d9/sub-test-model-fbb6d9-kserve-7548855b8f-k7dnz" event={"ID":"545773bc-bdc3-4b7d-8a44-af57a9c5a368","Type":"ContainerStarted","Data":"35eaf1c5fb5d30e835befb8a51e2ae3a599d06d02b4a63de957f443e918c6dad"} Sep 12 02:13:03.801320 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:03.801266 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-fbb6d9/sub-test-model-fbb6d9-kserve-7548855b8f-k7dnz" podStartSLOduration=1.8012513650000002 podStartE2EDuration="1.801251365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:13:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:13:03.799713378 +0000 UTC m=+1686.591707173" watchObservedRunningTime="2026-09-12 02:13:03.801251365 +0000 UTC m=+1686.593245160" Sep 12 02:13:05.925322 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:05.925283 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:13:08.465188 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:08.465148 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:13:12.744259 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:12.744207 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-fbb6d9/sub-test-model-fbb6d9-kserve-7548855b8f-k7dnz" Sep 12 02:13:12.744777 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:12.744352 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-fbb6d9/sub-test-model-fbb6d9-kserve-7548855b8f-k7dnz" Sep 12 02:13:12.756928 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:12.756905 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-fbb6d9/sub-test-model-fbb6d9-kserve-7548855b8f-k7dnz" Sep 12 02:13:12.842399 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:12.842372 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-fbb6d9/sub-test-model-fbb6d9-kserve-7548855b8f-k7dnz" Sep 12 02:13:28.820716 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:28.820641 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:13:30.427335 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:30.427299 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:13:30.527360 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:30.527328 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:13:44.821978 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:44.821940 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:13:45.530217 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:45.530178 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:13:46.403347 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:46.403317 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-fbb6d9/sub-test-model-fbb6d9-kserve-7548855b8f-k7dnz"] Sep 12 02:13:46.422709 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:46.422677 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:13:53.017183 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:13:53.017147 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:14:01.423534 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:01.423488 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-fbb6d9/sub-test-model-fbb6d9-kserve-7548855b8f-k7dnz" podUID="545773bc-bdc3-4b7d-8a44-af57a9c5a368" containerName="main" containerID="cri-o://025066ed600a1b44851afeeb4dbc2c4b75f1fa41e50aba1f60219101a8f9ebbd" gracePeriod=45 Sep 12 02:14:01.680010 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:01.679946 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-fbb6d9/sub-test-model-fbb6d9-kserve-7548855b8f-k7dnz" Sep 12 02:14:01.776412 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:01.776372 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/545773bc-bdc3-4b7d-8a44-af57a9c5a368-model-cache\") pod \"545773bc-bdc3-4b7d-8a44-af57a9c5a368\" (UID: \"545773bc-bdc3-4b7d-8a44-af57a9c5a368\") " Sep 12 02:14:01.776412 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:01.776408 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/545773bc-bdc3-4b7d-8a44-af57a9c5a368-home\") pod \"545773bc-bdc3-4b7d-8a44-af57a9c5a368\" (UID: \"545773bc-bdc3-4b7d-8a44-af57a9c5a368\") " Sep 12 02:14:01.776702 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:01.776437 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fxdck\" (UniqueName: \"kubernetes.io/projected/545773bc-bdc3-4b7d-8a44-af57a9c5a368-kube-api-access-fxdck\") pod \"545773bc-bdc3-4b7d-8a44-af57a9c5a368\" (UID: \"545773bc-bdc3-4b7d-8a44-af57a9c5a368\") " Sep 12 02:14:01.776702 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:01.776509 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/545773bc-bdc3-4b7d-8a44-af57a9c5a368-tls-certs\") pod \"545773bc-bdc3-4b7d-8a44-af57a9c5a368\" (UID: \"545773bc-bdc3-4b7d-8a44-af57a9c5a368\") " Sep 12 02:14:01.776702 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:01.776550 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/545773bc-bdc3-4b7d-8a44-af57a9c5a368-tmp-dir\") pod \"545773bc-bdc3-4b7d-8a44-af57a9c5a368\" (UID: \"545773bc-bdc3-4b7d-8a44-af57a9c5a368\") " Sep 12 02:14:01.776702 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:01.776578 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/545773bc-bdc3-4b7d-8a44-af57a9c5a368-dshm\") pod \"545773bc-bdc3-4b7d-8a44-af57a9c5a368\" (UID: \"545773bc-bdc3-4b7d-8a44-af57a9c5a368\") " Sep 12 02:14:01.776912 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:01.776713 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/545773bc-bdc3-4b7d-8a44-af57a9c5a368-model-cache" (OuterVolumeSpecName: "model-cache") pod "545773bc-bdc3-4b7d-8a44-af57a9c5a368" (UID: "545773bc-bdc3-4b7d-8a44-af57a9c5a368"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 12 02:14:01.776912 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:01.776732 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/545773bc-bdc3-4b7d-8a44-af57a9c5a368-home" (OuterVolumeSpecName: "home") pod "545773bc-bdc3-4b7d-8a44-af57a9c5a368" (UID: "545773bc-bdc3-4b7d-8a44-af57a9c5a368"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 12 02:14:01.777016 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:01.776957 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/545773bc-bdc3-4b7d-8a44-af57a9c5a368-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "545773bc-bdc3-4b7d-8a44-af57a9c5a368" (UID: "545773bc-bdc3-4b7d-8a44-af57a9c5a368"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 12 02:14:01.777016 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:01.776966 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/545773bc-bdc3-4b7d-8a44-af57a9c5a368-model-cache\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:14:01.777016 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:01.776985 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/545773bc-bdc3-4b7d-8a44-af57a9c5a368-home\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:14:01.778954 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:01.778929 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/545773bc-bdc3-4b7d-8a44-af57a9c5a368-dshm" (OuterVolumeSpecName: "dshm") pod "545773bc-bdc3-4b7d-8a44-af57a9c5a368" (UID: "545773bc-bdc3-4b7d-8a44-af57a9c5a368"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 12 02:14:01.779032 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:01.778935 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/545773bc-bdc3-4b7d-8a44-af57a9c5a368-kube-api-access-fxdck" (OuterVolumeSpecName: "kube-api-access-fxdck") pod "545773bc-bdc3-4b7d-8a44-af57a9c5a368" (UID: "545773bc-bdc3-4b7d-8a44-af57a9c5a368"). InnerVolumeSpecName "kube-api-access-fxdck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 12 02:14:01.779032 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:01.778989 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/545773bc-bdc3-4b7d-8a44-af57a9c5a368-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "545773bc-bdc3-4b7d-8a44-af57a9c5a368" (UID: "545773bc-bdc3-4b7d-8a44-af57a9c5a368"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 12 02:14:01.877969 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:01.877915 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/545773bc-bdc3-4b7d-8a44-af57a9c5a368-tls-certs\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:14:01.877969 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:01.877966 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/545773bc-bdc3-4b7d-8a44-af57a9c5a368-tmp-dir\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:14:01.877969 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:01.877979 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/545773bc-bdc3-4b7d-8a44-af57a9c5a368-dshm\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:14:01.878212 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:01.877993 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxdck\" (UniqueName: \"kubernetes.io/projected/545773bc-bdc3-4b7d-8a44-af57a9c5a368-kube-api-access-fxdck\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:14:02.075440 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:02.075347 2579 generic.go:356] "Generic (PLEG): container finished" podID="545773bc-bdc3-4b7d-8a44-af57a9c5a368" containerID="025066ed600a1b44851afeeb4dbc2c4b75f1fa41e50aba1f60219101a8f9ebbd" exitCode=0 Sep 12 02:14:02.075440 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:02.075400 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-fbb6d9/sub-test-model-fbb6d9-kserve-7548855b8f-k7dnz" Sep 12 02:14:02.075686 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:02.075404 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-fbb6d9/sub-test-model-fbb6d9-kserve-7548855b8f-k7dnz" event={"ID":"545773bc-bdc3-4b7d-8a44-af57a9c5a368","Type":"ContainerDied","Data":"025066ed600a1b44851afeeb4dbc2c4b75f1fa41e50aba1f60219101a8f9ebbd"} Sep 12 02:14:02.075686 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:02.075509 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-fbb6d9/sub-test-model-fbb6d9-kserve-7548855b8f-k7dnz" event={"ID":"545773bc-bdc3-4b7d-8a44-af57a9c5a368","Type":"ContainerDied","Data":"35eaf1c5fb5d30e835befb8a51e2ae3a599d06d02b4a63de957f443e918c6dad"} Sep 12 02:14:02.075686 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:02.075534 2579 scope.go:117] "RemoveContainer" containerID="025066ed600a1b44851afeeb4dbc2c4b75f1fa41e50aba1f60219101a8f9ebbd" Sep 12 02:14:02.087408 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:02.087388 2579 scope.go:117] "RemoveContainer" containerID="025066ed600a1b44851afeeb4dbc2c4b75f1fa41e50aba1f60219101a8f9ebbd" Sep 12 02:14:02.087789 ip-10-0-133-79 kubenswrapper[2579]: E0912 02:14:02.087767 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"025066ed600a1b44851afeeb4dbc2c4b75f1fa41e50aba1f60219101a8f9ebbd\": container with ID starting with 025066ed600a1b44851afeeb4dbc2c4b75f1fa41e50aba1f60219101a8f9ebbd not found: ID does not exist" containerID="025066ed600a1b44851afeeb4dbc2c4b75f1fa41e50aba1f60219101a8f9ebbd" Sep 12 02:14:02.087861 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:02.087804 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"025066ed600a1b44851afeeb4dbc2c4b75f1fa41e50aba1f60219101a8f9ebbd"} err="failed to get container status \"025066ed600a1b44851afeeb4dbc2c4b75f1fa41e50aba1f60219101a8f9ebbd\": rpc error: code = NotFound desc = could not find container \"025066ed600a1b44851afeeb4dbc2c4b75f1fa41e50aba1f60219101a8f9ebbd\": container with ID starting with 025066ed600a1b44851afeeb4dbc2c4b75f1fa41e50aba1f60219101a8f9ebbd not found: ID does not exist" Sep 12 02:14:02.097878 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:02.097841 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-fbb6d9/sub-test-model-fbb6d9-kserve-7548855b8f-k7dnz"] Sep 12 02:14:02.101282 ip-10-0-133-79 kubenswrapper[2579]: E0912 02:14:02.101244 2579 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod545773bc_bdc3_4b7d_8a44_af57a9c5a368.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod545773bc_bdc3_4b7d_8a44_af57a9c5a368.slice/crio-35eaf1c5fb5d30e835befb8a51e2ae3a599d06d02b4a63de957f443e918c6dad\": RecentStats: unable to find data in memory cache]" Sep 12 02:14:02.101412 ip-10-0-133-79 kubenswrapper[2579]: E0912 02:14:02.101253 2579 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod545773bc_bdc3_4b7d_8a44_af57a9c5a368.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod545773bc_bdc3_4b7d_8a44_af57a9c5a368.slice/crio-35eaf1c5fb5d30e835befb8a51e2ae3a599d06d02b4a63de957f443e918c6dad\": RecentStats: unable to find data in memory cache]" Sep 12 02:14:02.102516 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:02.102498 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-fbb6d9/sub-test-model-fbb6d9-kserve-7548855b8f-k7dnz"] Sep 12 02:14:03.553252 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:03.553220 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:14:03.727574 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:03.727541 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="545773bc-bdc3-4b7d-8a44-af57a9c5a368" path="/var/lib/kubelet/pods/545773bc-bdc3-4b7d-8a44-af57a9c5a368/volumes" Sep 12 02:14:06.727605 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:06.727569 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:14:09.050010 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:09.049975 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:14:09.150856 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:09.150822 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:14:42.823012 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:42.822978 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-fbb6d9/rate-test-model-fbb6d9-kserve-6994775775-5pj9l"] Sep 12 02:14:42.823493 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:42.823410 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="545773bc-bdc3-4b7d-8a44-af57a9c5a368" containerName="main" Sep 12 02:14:42.823493 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:42.823421 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="545773bc-bdc3-4b7d-8a44-af57a9c5a368" containerName="main" Sep 12 02:14:42.823577 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:42.823512 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="545773bc-bdc3-4b7d-8a44-af57a9c5a368" containerName="main" Sep 12 02:14:42.826463 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:42.826435 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-fbb6d9/rate-test-model-fbb6d9-kserve-6994775775-5pj9l" Sep 12 02:14:42.829900 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:42.829869 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-fbb6d9\"/\"default-dockercfg-mjwfn\"" Sep 12 02:14:42.830064 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:42.829945 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-fbb6d9\"/\"rate-test-model-fbb6d9-kserve-self-signed-certs\"" Sep 12 02:14:42.830064 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:42.829972 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-fbb6d9\"/\"kube-root-ca.crt\"" Sep 12 02:14:42.830064 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:42.829986 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-fbb6d9\"/\"openshift-service-ca.crt\"" Sep 12 02:14:42.836199 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:42.836159 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-fbb6d9/rate-test-model-fbb6d9-kserve-6994775775-5pj9l"] Sep 12 02:14:42.853237 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:42.853193 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:14:42.975629 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:42.975588 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkkgd\" (UniqueName: \"kubernetes.io/projected/bf6ba24c-5a52-41b1-b000-3e07ae47c0dc-kube-api-access-vkkgd\") pod \"rate-test-model-fbb6d9-kserve-6994775775-5pj9l\" (UID: \"bf6ba24c-5a52-41b1-b000-3e07ae47c0dc\") " pod="ai-tenant-e2e-shared-b-w3-fbb6d9/rate-test-model-fbb6d9-kserve-6994775775-5pj9l" Sep 12 02:14:42.975629 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:42.975632 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf6ba24c-5a52-41b1-b000-3e07ae47c0dc-home\") pod \"rate-test-model-fbb6d9-kserve-6994775775-5pj9l\" (UID: \"bf6ba24c-5a52-41b1-b000-3e07ae47c0dc\") " pod="ai-tenant-e2e-shared-b-w3-fbb6d9/rate-test-model-fbb6d9-kserve-6994775775-5pj9l" Sep 12 02:14:42.975861 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:42.975666 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf6ba24c-5a52-41b1-b000-3e07ae47c0dc-dshm\") pod \"rate-test-model-fbb6d9-kserve-6994775775-5pj9l\" (UID: \"bf6ba24c-5a52-41b1-b000-3e07ae47c0dc\") " pod="ai-tenant-e2e-shared-b-w3-fbb6d9/rate-test-model-fbb6d9-kserve-6994775775-5pj9l" Sep 12 02:14:42.975861 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:42.975730 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf6ba24c-5a52-41b1-b000-3e07ae47c0dc-model-cache\") pod \"rate-test-model-fbb6d9-kserve-6994775775-5pj9l\" (UID: \"bf6ba24c-5a52-41b1-b000-3e07ae47c0dc\") " pod="ai-tenant-e2e-shared-b-w3-fbb6d9/rate-test-model-fbb6d9-kserve-6994775775-5pj9l" Sep 12 02:14:42.975861 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:42.975783 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf6ba24c-5a52-41b1-b000-3e07ae47c0dc-tls-certs\") pod \"rate-test-model-fbb6d9-kserve-6994775775-5pj9l\" (UID: \"bf6ba24c-5a52-41b1-b000-3e07ae47c0dc\") " pod="ai-tenant-e2e-shared-b-w3-fbb6d9/rate-test-model-fbb6d9-kserve-6994775775-5pj9l" Sep 12 02:14:42.975861 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:42.975852 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf6ba24c-5a52-41b1-b000-3e07ae47c0dc-tmp-dir\") pod \"rate-test-model-fbb6d9-kserve-6994775775-5pj9l\" (UID: \"bf6ba24c-5a52-41b1-b000-3e07ae47c0dc\") " pod="ai-tenant-e2e-shared-b-w3-fbb6d9/rate-test-model-fbb6d9-kserve-6994775775-5pj9l" Sep 12 02:14:43.077400 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:43.077309 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkkgd\" (UniqueName: \"kubernetes.io/projected/bf6ba24c-5a52-41b1-b000-3e07ae47c0dc-kube-api-access-vkkgd\") pod \"rate-test-model-fbb6d9-kserve-6994775775-5pj9l\" (UID: \"bf6ba24c-5a52-41b1-b000-3e07ae47c0dc\") " pod="ai-tenant-e2e-shared-b-w3-fbb6d9/rate-test-model-fbb6d9-kserve-6994775775-5pj9l" Sep 12 02:14:43.077400 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:43.077346 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf6ba24c-5a52-41b1-b000-3e07ae47c0dc-home\") pod \"rate-test-model-fbb6d9-kserve-6994775775-5pj9l\" (UID: \"bf6ba24c-5a52-41b1-b000-3e07ae47c0dc\") " pod="ai-tenant-e2e-shared-b-w3-fbb6d9/rate-test-model-fbb6d9-kserve-6994775775-5pj9l" Sep 12 02:14:43.077400 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:43.077379 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf6ba24c-5a52-41b1-b000-3e07ae47c0dc-dshm\") pod \"rate-test-model-fbb6d9-kserve-6994775775-5pj9l\" (UID: \"bf6ba24c-5a52-41b1-b000-3e07ae47c0dc\") " pod="ai-tenant-e2e-shared-b-w3-fbb6d9/rate-test-model-fbb6d9-kserve-6994775775-5pj9l" Sep 12 02:14:43.077707 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:43.077416 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf6ba24c-5a52-41b1-b000-3e07ae47c0dc-model-cache\") pod \"rate-test-model-fbb6d9-kserve-6994775775-5pj9l\" (UID: \"bf6ba24c-5a52-41b1-b000-3e07ae47c0dc\") " pod="ai-tenant-e2e-shared-b-w3-fbb6d9/rate-test-model-fbb6d9-kserve-6994775775-5pj9l" Sep 12 02:14:43.077707 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:43.077495 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf6ba24c-5a52-41b1-b000-3e07ae47c0dc-tls-certs\") pod \"rate-test-model-fbb6d9-kserve-6994775775-5pj9l\" (UID: \"bf6ba24c-5a52-41b1-b000-3e07ae47c0dc\") " pod="ai-tenant-e2e-shared-b-w3-fbb6d9/rate-test-model-fbb6d9-kserve-6994775775-5pj9l" Sep 12 02:14:43.077707 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:43.077534 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf6ba24c-5a52-41b1-b000-3e07ae47c0dc-tmp-dir\") pod \"rate-test-model-fbb6d9-kserve-6994775775-5pj9l\" (UID: \"bf6ba24c-5a52-41b1-b000-3e07ae47c0dc\") " pod="ai-tenant-e2e-shared-b-w3-fbb6d9/rate-test-model-fbb6d9-kserve-6994775775-5pj9l" Sep 12 02:14:43.077866 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:43.077832 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf6ba24c-5a52-41b1-b000-3e07ae47c0dc-home\") pod \"rate-test-model-fbb6d9-kserve-6994775775-5pj9l\" (UID: \"bf6ba24c-5a52-41b1-b000-3e07ae47c0dc\") " pod="ai-tenant-e2e-shared-b-w3-fbb6d9/rate-test-model-fbb6d9-kserve-6994775775-5pj9l" Sep 12 02:14:43.077866 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:43.077844 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf6ba24c-5a52-41b1-b000-3e07ae47c0dc-tmp-dir\") pod \"rate-test-model-fbb6d9-kserve-6994775775-5pj9l\" (UID: \"bf6ba24c-5a52-41b1-b000-3e07ae47c0dc\") " pod="ai-tenant-e2e-shared-b-w3-fbb6d9/rate-test-model-fbb6d9-kserve-6994775775-5pj9l" Sep 12 02:14:43.077943 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:43.077910 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf6ba24c-5a52-41b1-b000-3e07ae47c0dc-model-cache\") pod \"rate-test-model-fbb6d9-kserve-6994775775-5pj9l\" (UID: \"bf6ba24c-5a52-41b1-b000-3e07ae47c0dc\") " pod="ai-tenant-e2e-shared-b-w3-fbb6d9/rate-test-model-fbb6d9-kserve-6994775775-5pj9l" Sep 12 02:14:43.079854 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:43.079834 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf6ba24c-5a52-41b1-b000-3e07ae47c0dc-dshm\") pod \"rate-test-model-fbb6d9-kserve-6994775775-5pj9l\" (UID: \"bf6ba24c-5a52-41b1-b000-3e07ae47c0dc\") " pod="ai-tenant-e2e-shared-b-w3-fbb6d9/rate-test-model-fbb6d9-kserve-6994775775-5pj9l" Sep 12 02:14:43.080105 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:43.080090 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf6ba24c-5a52-41b1-b000-3e07ae47c0dc-tls-certs\") pod \"rate-test-model-fbb6d9-kserve-6994775775-5pj9l\" (UID: \"bf6ba24c-5a52-41b1-b000-3e07ae47c0dc\") " pod="ai-tenant-e2e-shared-b-w3-fbb6d9/rate-test-model-fbb6d9-kserve-6994775775-5pj9l" Sep 12 02:14:43.085018 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:43.084996 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkkgd\" (UniqueName: \"kubernetes.io/projected/bf6ba24c-5a52-41b1-b000-3e07ae47c0dc-kube-api-access-vkkgd\") pod \"rate-test-model-fbb6d9-kserve-6994775775-5pj9l\" (UID: \"bf6ba24c-5a52-41b1-b000-3e07ae47c0dc\") " pod="ai-tenant-e2e-shared-b-w3-fbb6d9/rate-test-model-fbb6d9-kserve-6994775775-5pj9l" Sep 12 02:14:43.138754 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:43.138722 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-fbb6d9/rate-test-model-fbb6d9-kserve-6994775775-5pj9l" Sep 12 02:14:43.408655 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:43.408630 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-fbb6d9/rate-test-model-fbb6d9-kserve-6994775775-5pj9l"] Sep 12 02:14:43.410620 ip-10-0-133-79 kubenswrapper[2579]: W0912 02:14:43.410592 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf6ba24c_5a52_41b1_b000_3e07ae47c0dc.slice/crio-2fea47b5326741fc2da0f89a2a90a6a1f082e12d580304d668da4fc633e72230 WatchSource:0}: Error finding container 2fea47b5326741fc2da0f89a2a90a6a1f082e12d580304d668da4fc633e72230: Status 404 returned error can't find the container with id 2fea47b5326741fc2da0f89a2a90a6a1f082e12d580304d668da4fc633e72230 Sep 12 02:14:44.291872 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:44.291833 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-fbb6d9/rate-test-model-fbb6d9-kserve-6994775775-5pj9l" event={"ID":"bf6ba24c-5a52-41b1-b000-3e07ae47c0dc","Type":"ContainerStarted","Data":"4ce14cc4386b4712bd02b66f50f4c71e7f91ca88399991142e65cc7208654044"} Sep 12 02:14:44.291872 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:44.291877 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-fbb6d9/rate-test-model-fbb6d9-kserve-6994775775-5pj9l" event={"ID":"bf6ba24c-5a52-41b1-b000-3e07ae47c0dc","Type":"ContainerStarted","Data":"2fea47b5326741fc2da0f89a2a90a6a1f082e12d580304d668da4fc633e72230"} Sep 12 02:14:44.308760 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:44.308711 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-fbb6d9/rate-test-model-fbb6d9-kserve-6994775775-5pj9l" podStartSLOduration=2.3086945549999998 podStartE2EDuration="2.308694555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:14:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:14:44.306597095 +0000 UTC m=+1787.098590894" watchObservedRunningTime="2026-09-12 02:14:44.308694555 +0000 UTC m=+1787.100688349" Sep 12 02:14:46.230355 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:46.230316 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:14:50.461992 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:50.461905 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:14:50.523915 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:50.523877 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:14:53.139355 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:53.139311 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-fbb6d9/rate-test-model-fbb6d9-kserve-6994775775-5pj9l" Sep 12 02:14:53.139856 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:53.139369 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-fbb6d9/rate-test-model-fbb6d9-kserve-6994775775-5pj9l" Sep 12 02:14:53.152003 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:53.151967 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-fbb6d9/rate-test-model-fbb6d9-kserve-6994775775-5pj9l" Sep 12 02:14:53.352224 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:14:53.352182 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-fbb6d9/rate-test-model-fbb6d9-kserve-6994775775-5pj9l" Sep 12 02:15:09.626545 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:15:09.626510 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:15:24.420983 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:15:24.420946 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:15:30.323639 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:15:30.323596 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:15:31.311171 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:15:31.311135 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-fbb6d9/rate-test-model-fbb6d9-kserve-6994775775-5pj9l"] Sep 12 02:15:31.461569 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:15:31.461535 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:15:31.788816 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:15:31.788785 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:15:32.199106 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:15:32.199074 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:15:32.462825 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:15:32.462738 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:15:34.419385 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:15:34.419347 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:15:34.855715 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:15:34.855630 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:15:35.246547 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:15:35.246513 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:15:35.341646 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:15:35.341611 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:15:35.773814 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:15:35.773778 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:15:36.559942 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:15:36.559909 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:15:36.659205 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:15:36.659170 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:15:37.008380 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:15:37.008340 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:15:37.111798 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:15:37.111763 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:15:37.358993 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:15:37.358905 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:15:38.860237 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:15:38.860186 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:15:44.518272 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:15:44.518238 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:15:46.332634 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:15:46.332594 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-fbb6d9/rate-test-model-fbb6d9-kserve-6994775775-5pj9l" podUID="bf6ba24c-5a52-41b1-b000-3e07ae47c0dc" containerName="main" containerID="cri-o://4ce14cc4386b4712bd02b66f50f4c71e7f91ca88399991142e65cc7208654044" gracePeriod=45 Sep 12 02:15:46.593583 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:15:46.593519 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-fbb6d9/rate-test-model-fbb6d9-kserve-6994775775-5pj9l" Sep 12 02:15:46.601392 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:15:46.601365 2579 generic.go:356] "Generic (PLEG): container finished" podID="bf6ba24c-5a52-41b1-b000-3e07ae47c0dc" containerID="4ce14cc4386b4712bd02b66f50f4c71e7f91ca88399991142e65cc7208654044" exitCode=0 Sep 12 02:15:46.601563 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:15:46.601402 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-fbb6d9/rate-test-model-fbb6d9-kserve-6994775775-5pj9l" Sep 12 02:15:46.601563 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:15:46.601432 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-fbb6d9/rate-test-model-fbb6d9-kserve-6994775775-5pj9l" event={"ID":"bf6ba24c-5a52-41b1-b000-3e07ae47c0dc","Type":"ContainerDied","Data":"4ce14cc4386b4712bd02b66f50f4c71e7f91ca88399991142e65cc7208654044"} Sep 12 02:15:46.601563 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:15:46.601490 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-fbb6d9/rate-test-model-fbb6d9-kserve-6994775775-5pj9l" event={"ID":"bf6ba24c-5a52-41b1-b000-3e07ae47c0dc","Type":"ContainerDied","Data":"2fea47b5326741fc2da0f89a2a90a6a1f082e12d580304d668da4fc633e72230"} Sep 12 02:15:46.601563 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:15:46.601519 2579 scope.go:117] "RemoveContainer" containerID="4ce14cc4386b4712bd02b66f50f4c71e7f91ca88399991142e65cc7208654044" Sep 12 02:15:46.610546 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:15:46.610519 2579 scope.go:117] "RemoveContainer" containerID="4ce14cc4386b4712bd02b66f50f4c71e7f91ca88399991142e65cc7208654044" Sep 12 02:15:46.610942 ip-10-0-133-79 kubenswrapper[2579]: E0912 02:15:46.610915 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ce14cc4386b4712bd02b66f50f4c71e7f91ca88399991142e65cc7208654044\": container with ID starting with 4ce14cc4386b4712bd02b66f50f4c71e7f91ca88399991142e65cc7208654044 not found: ID does not exist" containerID="4ce14cc4386b4712bd02b66f50f4c71e7f91ca88399991142e65cc7208654044" Sep 12 02:15:46.611052 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:15:46.610953 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ce14cc4386b4712bd02b66f50f4c71e7f91ca88399991142e65cc7208654044"} err="failed to get container status \"4ce14cc4386b4712bd02b66f50f4c71e7f91ca88399991142e65cc7208654044\": rpc error: code = NotFound desc = could not find container \"4ce14cc4386b4712bd02b66f50f4c71e7f91ca88399991142e65cc7208654044\": container with ID starting with 4ce14cc4386b4712bd02b66f50f4c71e7f91ca88399991142e65cc7208654044 not found: ID does not exist" Sep 12 02:15:46.693737 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:15:46.693704 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vkkgd\" (UniqueName: \"kubernetes.io/projected/bf6ba24c-5a52-41b1-b000-3e07ae47c0dc-kube-api-access-vkkgd\") pod \"bf6ba24c-5a52-41b1-b000-3e07ae47c0dc\" (UID: \"bf6ba24c-5a52-41b1-b000-3e07ae47c0dc\") " Sep 12 02:15:46.693931 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:15:46.693758 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf6ba24c-5a52-41b1-b000-3e07ae47c0dc-tmp-dir\") pod \"bf6ba24c-5a52-41b1-b000-3e07ae47c0dc\" (UID: \"bf6ba24c-5a52-41b1-b000-3e07ae47c0dc\") " Sep 12 02:15:46.693931 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:15:46.693810 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf6ba24c-5a52-41b1-b000-3e07ae47c0dc-home\") pod \"bf6ba24c-5a52-41b1-b000-3e07ae47c0dc\" (UID: \"bf6ba24c-5a52-41b1-b000-3e07ae47c0dc\") " Sep 12 02:15:46.693931 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:15:46.693837 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf6ba24c-5a52-41b1-b000-3e07ae47c0dc-model-cache\") pod \"bf6ba24c-5a52-41b1-b000-3e07ae47c0dc\" (UID: \"bf6ba24c-5a52-41b1-b000-3e07ae47c0dc\") " Sep 12 02:15:46.693931 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:15:46.693868 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf6ba24c-5a52-41b1-b000-3e07ae47c0dc-dshm\") pod \"bf6ba24c-5a52-41b1-b000-3e07ae47c0dc\" (UID: \"bf6ba24c-5a52-41b1-b000-3e07ae47c0dc\") " Sep 12 02:15:46.693931 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:15:46.693915 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf6ba24c-5a52-41b1-b000-3e07ae47c0dc-tls-certs\") pod \"bf6ba24c-5a52-41b1-b000-3e07ae47c0dc\" (UID: \"bf6ba24c-5a52-41b1-b000-3e07ae47c0dc\") " Sep 12 02:15:46.694189 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:15:46.694098 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf6ba24c-5a52-41b1-b000-3e07ae47c0dc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bf6ba24c-5a52-41b1-b000-3e07ae47c0dc" (UID: "bf6ba24c-5a52-41b1-b000-3e07ae47c0dc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 12 02:15:46.694189 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:15:46.694109 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf6ba24c-5a52-41b1-b000-3e07ae47c0dc-home" (OuterVolumeSpecName: "home") pod "bf6ba24c-5a52-41b1-b000-3e07ae47c0dc" (UID: "bf6ba24c-5a52-41b1-b000-3e07ae47c0dc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 12 02:15:46.694189 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:15:46.694123 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf6ba24c-5a52-41b1-b000-3e07ae47c0dc-model-cache" (OuterVolumeSpecName: "model-cache") pod "bf6ba24c-5a52-41b1-b000-3e07ae47c0dc" (UID: "bf6ba24c-5a52-41b1-b000-3e07ae47c0dc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 12 02:15:46.696142 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:15:46.696112 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf6ba24c-5a52-41b1-b000-3e07ae47c0dc-dshm" (OuterVolumeSpecName: "dshm") pod "bf6ba24c-5a52-41b1-b000-3e07ae47c0dc" (UID: "bf6ba24c-5a52-41b1-b000-3e07ae47c0dc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 12 02:15:46.696266 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:15:46.696203 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf6ba24c-5a52-41b1-b000-3e07ae47c0dc-kube-api-access-vkkgd" (OuterVolumeSpecName: "kube-api-access-vkkgd") pod "bf6ba24c-5a52-41b1-b000-3e07ae47c0dc" (UID: "bf6ba24c-5a52-41b1-b000-3e07ae47c0dc"). InnerVolumeSpecName "kube-api-access-vkkgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 12 02:15:46.696266 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:15:46.696215 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf6ba24c-5a52-41b1-b000-3e07ae47c0dc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bf6ba24c-5a52-41b1-b000-3e07ae47c0dc" (UID: "bf6ba24c-5a52-41b1-b000-3e07ae47c0dc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 12 02:15:46.795106 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:15:46.795050 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf6ba24c-5a52-41b1-b000-3e07ae47c0dc-tmp-dir\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:15:46.795106 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:15:46.795081 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf6ba24c-5a52-41b1-b000-3e07ae47c0dc-home\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:15:46.795106 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:15:46.795095 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf6ba24c-5a52-41b1-b000-3e07ae47c0dc-model-cache\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:15:46.795106 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:15:46.795110 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf6ba24c-5a52-41b1-b000-3e07ae47c0dc-dshm\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:15:46.795372 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:15:46.795121 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf6ba24c-5a52-41b1-b000-3e07ae47c0dc-tls-certs\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:15:46.795372 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:15:46.795135 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkkgd\" (UniqueName: \"kubernetes.io/projected/bf6ba24c-5a52-41b1-b000-3e07ae47c0dc-kube-api-access-vkkgd\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:15:46.923441 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:15:46.923411 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-fbb6d9/rate-test-model-fbb6d9-kserve-6994775775-5pj9l"] Sep 12 02:15:46.929214 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:15:46.929191 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-fbb6d9/rate-test-model-fbb6d9-kserve-6994775775-5pj9l"] Sep 12 02:15:47.725881 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:15:47.725841 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf6ba24c-5a52-41b1-b000-3e07ae47c0dc" path="/var/lib/kubelet/pods/bf6ba24c-5a52-41b1-b000-3e07ae47c0dc/volumes" Sep 12 02:15:53.189543 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:15:53.189508 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:15:58.467640 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:15:58.467593 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:15:59.626558 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:15:59.626523 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:16:00.187979 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:00.187938 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:16:00.365829 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:00.365792 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:16:00.923144 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:00.923108 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:16:01.174031 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:01.173936 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:16:01.238880 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:01.238851 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:16:01.377672 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:01.377633 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:16:01.858644 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:01.858607 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:16:04.030725 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:04.030684 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:16:04.293021 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:04.292934 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:16:04.386544 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:04.386498 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:16:04.861136 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:04.861096 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:16:05.633277 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:05.633237 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:16:06.350302 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:06.350267 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-01570e-59f8689478-tfnbh"] Sep 12 02:16:06.350923 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:06.350904 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf6ba24c-5a52-41b1-b000-3e07ae47c0dc" containerName="main" Sep 12 02:16:06.350923 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:06.350925 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf6ba24c-5a52-41b1-b000-3e07ae47c0dc" containerName="main" Sep 12 02:16:06.351065 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:06.351032 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf6ba24c-5a52-41b1-b000-3e07ae47c0dc" containerName="main" Sep 12 02:16:06.354746 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:06.354723 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01570e-59f8689478-tfnbh" Sep 12 02:16:06.357087 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:06.357068 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-01570e-serving-cert\"" Sep 12 02:16:06.369285 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:06.369257 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-01570e-59f8689478-tfnbh"] Sep 12 02:16:06.388500 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:06.388441 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:16:06.485970 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:06.485930 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-01570e-59f8689478-tfnbh\" (UID: \"d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01570e-59f8689478-tfnbh" Sep 12 02:16:06.486163 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:06.485984 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1-metrics-certs\") pod \"maas-api-e2e-ipp-b-01570e-59f8689478-tfnbh\" (UID: \"d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01570e-59f8689478-tfnbh" Sep 12 02:16:06.486163 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:06.486055 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgp28\" (UniqueName: \"kubernetes.io/projected/d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1-kube-api-access-tgp28\") pod \"maas-api-e2e-ipp-b-01570e-59f8689478-tfnbh\" (UID: \"d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01570e-59f8689478-tfnbh" Sep 12 02:16:06.486163 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:06.486105 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1-maas-api-tls\") pod \"maas-api-e2e-ipp-b-01570e-59f8689478-tfnbh\" (UID: \"d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01570e-59f8689478-tfnbh" Sep 12 02:16:06.488838 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:06.488810 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:16:06.586719 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:06.586681 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-01570e-59f8689478-tfnbh\" (UID: \"d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01570e-59f8689478-tfnbh" Sep 12 02:16:06.586955 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:06.586745 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1-metrics-certs\") pod \"maas-api-e2e-ipp-b-01570e-59f8689478-tfnbh\" (UID: \"d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01570e-59f8689478-tfnbh" Sep 12 02:16:06.586955 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:06.586787 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgp28\" (UniqueName: \"kubernetes.io/projected/d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1-kube-api-access-tgp28\") pod \"maas-api-e2e-ipp-b-01570e-59f8689478-tfnbh\" (UID: \"d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01570e-59f8689478-tfnbh" Sep 12 02:16:06.586955 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:06.586830 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1-maas-api-tls\") pod \"maas-api-e2e-ipp-b-01570e-59f8689478-tfnbh\" (UID: \"d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01570e-59f8689478-tfnbh" Sep 12 02:16:06.587080 ip-10-0-133-79 kubenswrapper[2579]: E0912 02:16:06.586978 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-01570e-serving-cert: secret "maas-api-e2e-ipp-b-01570e-serving-cert" not found Sep 12 02:16:06.587080 ip-10-0-133-79 kubenswrapper[2579]: E0912 02:16:06.587071 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1-maas-api-tls podName:d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1 nodeName:}" failed. No retries permitted until 2026-09-12 02:16:07.087048363 +0000 UTC m=+1869.879042161 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1-maas-api-tls") pod "maas-api-e2e-ipp-b-01570e-59f8689478-tfnbh" (UID: "d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1") : secret "maas-api-e2e-ipp-b-01570e-serving-cert" not found Sep 12 02:16:06.587400 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:06.587381 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-01570e-59f8689478-tfnbh\" (UID: \"d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01570e-59f8689478-tfnbh" Sep 12 02:16:06.589524 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:06.589502 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1-metrics-certs\") pod \"maas-api-e2e-ipp-b-01570e-59f8689478-tfnbh\" (UID: \"d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01570e-59f8689478-tfnbh" Sep 12 02:16:06.596347 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:06.596320 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgp28\" (UniqueName: \"kubernetes.io/projected/d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1-kube-api-access-tgp28\") pod \"maas-api-e2e-ipp-b-01570e-59f8689478-tfnbh\" (UID: \"d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01570e-59f8689478-tfnbh" Sep 12 02:16:06.637234 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:06.637186 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:16:06.841744 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:06.841708 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:16:06.933353 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:06.933260 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:16:06.938438 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:06.938407 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-01570e-665c84644b-f9zxk"] Sep 12 02:16:06.942336 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:06.942316 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01570e-665c84644b-f9zxk" Sep 12 02:16:06.944684 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:06.944658 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-01570e\"" Sep 12 02:16:06.944814 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:06.944684 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-01570e-dockercfg-lg92x\"" Sep 12 02:16:06.953724 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:06.953697 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-01570e-665c84644b-f9zxk"] Sep 12 02:16:07.092691 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:07.092650 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/719dd67d-9839-4be1-99fa-c51f22340e31-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-01570e-665c84644b-f9zxk\" (UID: \"719dd67d-9839-4be1-99fa-c51f22340e31\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01570e-665c84644b-f9zxk" Sep 12 02:16:07.092880 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:07.092718 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fbgf\" (UniqueName: \"kubernetes.io/projected/719dd67d-9839-4be1-99fa-c51f22340e31-kube-api-access-9fbgf\") pod \"payload-pre-processing-e2e-ipp-b-01570e-665c84644b-f9zxk\" (UID: \"719dd67d-9839-4be1-99fa-c51f22340e31\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01570e-665c84644b-f9zxk" Sep 12 02:16:07.092880 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:07.092766 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1-maas-api-tls\") pod \"maas-api-e2e-ipp-b-01570e-59f8689478-tfnbh\" (UID: \"d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01570e-59f8689478-tfnbh" Sep 12 02:16:07.095400 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:07.095365 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1-maas-api-tls\") pod \"maas-api-e2e-ipp-b-01570e-59f8689478-tfnbh\" (UID: \"d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01570e-59f8689478-tfnbh" Sep 12 02:16:07.194419 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:07.194327 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/719dd67d-9839-4be1-99fa-c51f22340e31-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-01570e-665c84644b-f9zxk\" (UID: \"719dd67d-9839-4be1-99fa-c51f22340e31\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01570e-665c84644b-f9zxk" Sep 12 02:16:07.194419 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:07.194388 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fbgf\" (UniqueName: \"kubernetes.io/projected/719dd67d-9839-4be1-99fa-c51f22340e31-kube-api-access-9fbgf\") pod \"payload-pre-processing-e2e-ipp-b-01570e-665c84644b-f9zxk\" (UID: \"719dd67d-9839-4be1-99fa-c51f22340e31\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01570e-665c84644b-f9zxk" Sep 12 02:16:07.195106 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:07.195084 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/719dd67d-9839-4be1-99fa-c51f22340e31-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-01570e-665c84644b-f9zxk\" (UID: \"719dd67d-9839-4be1-99fa-c51f22340e31\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01570e-665c84644b-f9zxk" Sep 12 02:16:07.202014 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:07.201996 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fbgf\" (UniqueName: \"kubernetes.io/projected/719dd67d-9839-4be1-99fa-c51f22340e31-kube-api-access-9fbgf\") pod \"payload-pre-processing-e2e-ipp-b-01570e-665c84644b-f9zxk\" (UID: \"719dd67d-9839-4be1-99fa-c51f22340e31\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01570e-665c84644b-f9zxk" Sep 12 02:16:07.254974 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:07.254942 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01570e-665c84644b-f9zxk" Sep 12 02:16:07.268131 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:07.268104 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01570e-59f8689478-tfnbh" Sep 12 02:16:07.529366 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:07.529330 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-01570e-665c84644b-f9zxk"] Sep 12 02:16:07.533777 ip-10-0-133-79 kubenswrapper[2579]: W0912 02:16:07.533742 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod719dd67d_9839_4be1_99fa_c51f22340e31.slice/crio-cab0876ef59d0ca5aa56a5d93d6448bcc4ca22c251eb9f0b33f3af9e19980063 WatchSource:0}: Error finding container cab0876ef59d0ca5aa56a5d93d6448bcc4ca22c251eb9f0b33f3af9e19980063: Status 404 returned error can't find the container with id cab0876ef59d0ca5aa56a5d93d6448bcc4ca22c251eb9f0b33f3af9e19980063 Sep 12 02:16:07.552662 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:07.552638 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-01570e-59f8689478-tfnbh"] Sep 12 02:16:07.554982 ip-10-0-133-79 kubenswrapper[2579]: W0912 02:16:07.554952 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2f52ed2_2d9b_40a0_88d7_0fe94aacdcd1.slice/crio-8aa353febdd27de324c5320b42b50770dec7dcaca906e9d474d3c335b7d7cd0d WatchSource:0}: Error finding container 8aa353febdd27de324c5320b42b50770dec7dcaca906e9d474d3c335b7d7cd0d: Status 404 returned error can't find the container with id 8aa353febdd27de324c5320b42b50770dec7dcaca906e9d474d3c335b7d7cd0d Sep 12 02:16:07.556501 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:07.556481 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 12 02:16:07.714799 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:07.714765 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01570e-59f8689478-tfnbh" event={"ID":"d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1","Type":"ContainerStarted","Data":"8aa353febdd27de324c5320b42b50770dec7dcaca906e9d474d3c335b7d7cd0d"} Sep 12 02:16:07.716416 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:07.716389 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01570e-665c84644b-f9zxk" event={"ID":"719dd67d-9839-4be1-99fa-c51f22340e31","Type":"ContainerStarted","Data":"021985d21265ca2cc1c811d0528e5950a85201f571d5c965633a218e3181a95f"} Sep 12 02:16:07.716563 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:07.716443 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01570e-665c84644b-f9zxk" event={"ID":"719dd67d-9839-4be1-99fa-c51f22340e31","Type":"ContainerStarted","Data":"cab0876ef59d0ca5aa56a5d93d6448bcc4ca22c251eb9f0b33f3af9e19980063"} Sep 12 02:16:07.716628 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:07.716582 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01570e-665c84644b-f9zxk" Sep 12 02:16:07.733833 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:07.733785 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01570e-665c84644b-f9zxk" podStartSLOduration=1.733770391 podStartE2EDuration="1.733770391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:16:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:16:07.732096485 +0000 UTC m=+1870.524090281" watchObservedRunningTime="2026-09-12 02:16:07.733770391 +0000 UTC m=+1870.525764186" Sep 12 02:16:08.722945 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:08.722906 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01570e-59f8689478-tfnbh" event={"ID":"d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1","Type":"ContainerStarted","Data":"adbd597855df1134d878204f7936688a16f4ed3043c21de3d51073f0668f8b67"} Sep 12 02:16:08.723437 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:08.723003 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01570e-59f8689478-tfnbh" Sep 12 02:16:08.740098 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:08.740041 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01570e-59f8689478-tfnbh" podStartSLOduration=2.389461918 podStartE2EDuration="2.740026634s" totalImagesPullingTime="350.564716ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:16:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:16:08.738117849 +0000 UTC m=+1871.530111645" watchObservedRunningTime="2026-09-12 02:16:08.740026634 +0000 UTC m=+1871.532020429" Sep 12 02:16:11.926937 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:11.926889 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:16:14.734406 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:14.734379 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01570e-59f8689478-tfnbh" Sep 12 02:16:18.725283 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:18.725189 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01570e-665c84644b-f9zxk" Sep 12 02:16:27.887160 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:27.887119 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:16:28.708332 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:28.708295 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:16:29.368686 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:29.368644 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:16:29.465954 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:29.465918 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:16:30.075182 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:30.075149 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:16:30.708975 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:30.708938 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:16:31.458459 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:31.458424 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:16:40.834201 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:40.834165 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:16:45.330862 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:16:45.330813 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:17:04.531378 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:04.531337 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:17:11.391298 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:11.391266 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:17:11.711949 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:11.711858 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:17:11.862944 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:11.862902 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:17:13.951392 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:13.951350 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:17:14.568279 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:14.568244 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:17:14.666853 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:14.666815 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:17:14.765917 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:14.765878 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:17:15.357110 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:15.357077 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:17:16.149973 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:16.149935 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:17:16.968683 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:16.968652 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:17:17.582907 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:17.582874 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-848a2c-56b778bdc6-d2mws"] Sep 12 02:17:17.586866 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:17.586851 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-848a2c-56b778bdc6-d2mws" Sep 12 02:17:17.589588 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:17.589544 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-848a2c-serving-cert\"" Sep 12 02:17:17.596304 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:17.596271 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-848a2c-56b778bdc6-d2mws"] Sep 12 02:17:17.637418 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:17.637382 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:17:17.690907 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:17.690880 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-848a2c-56b778bdc6-d2mws\" (UID: \"7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-848a2c-56b778bdc6-d2mws" Sep 12 02:17:17.691072 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:17.690927 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j658l\" (UniqueName: \"kubernetes.io/projected/7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b-kube-api-access-j658l\") pod \"maas-api-e2e-ipp-cleanup-848a2c-56b778bdc6-d2mws\" (UID: \"7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-848a2c-56b778bdc6-d2mws" Sep 12 02:17:17.691072 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:17.691037 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-848a2c-56b778bdc6-d2mws\" (UID: \"7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-848a2c-56b778bdc6-d2mws" Sep 12 02:17:17.691072 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:17.691064 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-848a2c-56b778bdc6-d2mws\" (UID: \"7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-848a2c-56b778bdc6-d2mws" Sep 12 02:17:17.737365 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:17.737325 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:17:17.791931 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:17.791902 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-848a2c-56b778bdc6-d2mws\" (UID: \"7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-848a2c-56b778bdc6-d2mws" Sep 12 02:17:17.791931 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:17.791933 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-848a2c-56b778bdc6-d2mws\" (UID: \"7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-848a2c-56b778bdc6-d2mws" Sep 12 02:17:17.792146 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:17.791999 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-848a2c-56b778bdc6-d2mws\" (UID: \"7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-848a2c-56b778bdc6-d2mws" Sep 12 02:17:17.792146 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:17.792020 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j658l\" (UniqueName: \"kubernetes.io/projected/7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b-kube-api-access-j658l\") pod \"maas-api-e2e-ipp-cleanup-848a2c-56b778bdc6-d2mws\" (UID: \"7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-848a2c-56b778bdc6-d2mws" Sep 12 02:17:17.792146 ip-10-0-133-79 kubenswrapper[2579]: E0912 02:17:17.792120 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-848a2c-serving-cert: secret "maas-api-e2e-ipp-cleanup-848a2c-serving-cert" not found Sep 12 02:17:17.792330 ip-10-0-133-79 kubenswrapper[2579]: E0912 02:17:17.792192 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b-maas-api-tls podName:7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b nodeName:}" failed. No retries permitted until 2026-09-12 02:17:18.292174504 +0000 UTC m=+1941.084168277 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-848a2c-56b778bdc6-d2mws" (UID: "7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b") : secret "maas-api-e2e-ipp-cleanup-848a2c-serving-cert" not found Sep 12 02:17:17.792675 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:17.792654 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-848a2c-56b778bdc6-d2mws\" (UID: \"7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-848a2c-56b778bdc6-d2mws" Sep 12 02:17:17.794541 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:17.794521 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-848a2c-56b778bdc6-d2mws\" (UID: \"7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-848a2c-56b778bdc6-d2mws" Sep 12 02:17:17.800298 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:17.800267 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j658l\" (UniqueName: \"kubernetes.io/projected/7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b-kube-api-access-j658l\") pod \"maas-api-e2e-ipp-cleanup-848a2c-56b778bdc6-d2mws\" (UID: \"7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-848a2c-56b778bdc6-d2mws" Sep 12 02:17:18.007268 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:18.007236 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:17:18.147483 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:18.147431 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:17:18.237418 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:18.237380 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:17:18.237418 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:18.237406 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-848a2c-55bfd6d9c4-6gkjr"] Sep 12 02:17:18.241142 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:18.241127 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-848a2c-55bfd6d9c4-6gkjr" Sep 12 02:17:18.243572 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:18.243554 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-848a2c-dockercfg-82zvd\"" Sep 12 02:17:18.243728 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:18.243712 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-848a2c\"" Sep 12 02:17:18.251484 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:18.251443 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-848a2c-55bfd6d9c4-6gkjr"] Sep 12 02:17:18.288436 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:18.288349 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-848a2c-5dd6c745f9-62t4h"] Sep 12 02:17:18.293510 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:18.293491 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-848a2c-5dd6c745f9-62t4h" Sep 12 02:17:18.297160 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:18.297141 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-848a2c-56b778bdc6-d2mws\" (UID: \"7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-848a2c-56b778bdc6-d2mws" Sep 12 02:17:18.300017 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:18.299970 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-848a2c-56b778bdc6-d2mws\" (UID: \"7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-848a2c-56b778bdc6-d2mws" Sep 12 02:17:18.302705 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:18.302687 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-848a2c-5dd6c745f9-62t4h"] Sep 12 02:17:18.398161 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:18.398131 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnrss\" (UniqueName: \"kubernetes.io/projected/a0906a1c-e691-4cc0-a14a-f9fafb3b6138-kube-api-access-bnrss\") pod \"payload-pre-processing-e2e-ipp-cleanup-848a2c-55bfd6d9c4-6gkjr\" (UID: \"a0906a1c-e691-4cc0-a14a-f9fafb3b6138\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-848a2c-55bfd6d9c4-6gkjr" Sep 12 02:17:18.398310 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:18.398180 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25c8342a-cc78-4226-84c1-1c3e4a2bc018-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-848a2c-5dd6c745f9-62t4h\" (UID: \"25c8342a-cc78-4226-84c1-1c3e4a2bc018\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-848a2c-5dd6c745f9-62t4h" Sep 12 02:17:18.398310 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:18.398260 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz6rg\" (UniqueName: \"kubernetes.io/projected/25c8342a-cc78-4226-84c1-1c3e4a2bc018-kube-api-access-nz6rg\") pod \"payload-processing-e2e-ipp-cleanup-848a2c-5dd6c745f9-62t4h\" (UID: \"25c8342a-cc78-4226-84c1-1c3e4a2bc018\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-848a2c-5dd6c745f9-62t4h" Sep 12 02:17:18.398310 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:18.398303 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0906a1c-e691-4cc0-a14a-f9fafb3b6138-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-848a2c-55bfd6d9c4-6gkjr\" (UID: \"a0906a1c-e691-4cc0-a14a-f9fafb3b6138\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-848a2c-55bfd6d9c4-6gkjr" Sep 12 02:17:18.499861 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:18.499828 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0906a1c-e691-4cc0-a14a-f9fafb3b6138-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-848a2c-55bfd6d9c4-6gkjr\" (UID: \"a0906a1c-e691-4cc0-a14a-f9fafb3b6138\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-848a2c-55bfd6d9c4-6gkjr" Sep 12 02:17:18.500015 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:18.499946 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnrss\" (UniqueName: \"kubernetes.io/projected/a0906a1c-e691-4cc0-a14a-f9fafb3b6138-kube-api-access-bnrss\") pod \"payload-pre-processing-e2e-ipp-cleanup-848a2c-55bfd6d9c4-6gkjr\" (UID: \"a0906a1c-e691-4cc0-a14a-f9fafb3b6138\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-848a2c-55bfd6d9c4-6gkjr" Sep 12 02:17:18.500015 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:18.499985 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25c8342a-cc78-4226-84c1-1c3e4a2bc018-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-848a2c-5dd6c745f9-62t4h\" (UID: \"25c8342a-cc78-4226-84c1-1c3e4a2bc018\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-848a2c-5dd6c745f9-62t4h" Sep 12 02:17:18.500093 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:18.500022 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz6rg\" (UniqueName: \"kubernetes.io/projected/25c8342a-cc78-4226-84c1-1c3e4a2bc018-kube-api-access-nz6rg\") pod \"payload-processing-e2e-ipp-cleanup-848a2c-5dd6c745f9-62t4h\" (UID: \"25c8342a-cc78-4226-84c1-1c3e4a2bc018\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-848a2c-5dd6c745f9-62t4h" Sep 12 02:17:18.500582 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:18.500564 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0906a1c-e691-4cc0-a14a-f9fafb3b6138-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-848a2c-55bfd6d9c4-6gkjr\" (UID: \"a0906a1c-e691-4cc0-a14a-f9fafb3b6138\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-848a2c-55bfd6d9c4-6gkjr" Sep 12 02:17:18.500661 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:18.500645 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25c8342a-cc78-4226-84c1-1c3e4a2bc018-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-848a2c-5dd6c745f9-62t4h\" (UID: \"25c8342a-cc78-4226-84c1-1c3e4a2bc018\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-848a2c-5dd6c745f9-62t4h" Sep 12 02:17:18.500715 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:18.500703 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-848a2c-56b778bdc6-d2mws" Sep 12 02:17:18.511394 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:18.511368 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnrss\" (UniqueName: \"kubernetes.io/projected/a0906a1c-e691-4cc0-a14a-f9fafb3b6138-kube-api-access-bnrss\") pod \"payload-pre-processing-e2e-ipp-cleanup-848a2c-55bfd6d9c4-6gkjr\" (UID: \"a0906a1c-e691-4cc0-a14a-f9fafb3b6138\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-848a2c-55bfd6d9c4-6gkjr" Sep 12 02:17:18.511501 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:18.511392 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz6rg\" (UniqueName: \"kubernetes.io/projected/25c8342a-cc78-4226-84c1-1c3e4a2bc018-kube-api-access-nz6rg\") pod \"payload-processing-e2e-ipp-cleanup-848a2c-5dd6c745f9-62t4h\" (UID: \"25c8342a-cc78-4226-84c1-1c3e4a2bc018\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-848a2c-5dd6c745f9-62t4h" Sep 12 02:17:18.553277 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:18.553246 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-848a2c-55bfd6d9c4-6gkjr" Sep 12 02:17:18.613876 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:18.613812 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-848a2c-5dd6c745f9-62t4h" Sep 12 02:17:18.773503 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:18.773478 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-848a2c-56b778bdc6-d2mws"] Sep 12 02:17:18.775028 ip-10-0-133-79 kubenswrapper[2579]: W0912 02:17:18.775000 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c1f5068_b488_4fc4_bac0_bb9c0cb41a9b.slice/crio-b62caa58e312234e47698b80a9e1042018fd500e72ae9ada38cadee57dccdfb6 WatchSource:0}: Error finding container b62caa58e312234e47698b80a9e1042018fd500e72ae9ada38cadee57dccdfb6: Status 404 returned error can't find the container with id b62caa58e312234e47698b80a9e1042018fd500e72ae9ada38cadee57dccdfb6 Sep 12 02:17:18.831963 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:18.831940 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-848a2c-55bfd6d9c4-6gkjr"] Sep 12 02:17:18.834045 ip-10-0-133-79 kubenswrapper[2579]: W0912 02:17:18.834018 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0906a1c_e691_4cc0_a14a_f9fafb3b6138.slice/crio-8f37ce9b2bb514d14732a0a3842d52de896216035cb9bff152fe2a2eabee5f41 WatchSource:0}: Error finding container 8f37ce9b2bb514d14732a0a3842d52de896216035cb9bff152fe2a2eabee5f41: Status 404 returned error can't find the container with id 8f37ce9b2bb514d14732a0a3842d52de896216035cb9bff152fe2a2eabee5f41 Sep 12 02:17:18.889601 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:18.889577 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-848a2c-5dd6c745f9-62t4h"] Sep 12 02:17:18.891478 ip-10-0-133-79 kubenswrapper[2579]: W0912 02:17:18.891433 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25c8342a_cc78_4226_84c1_1c3e4a2bc018.slice/crio-f0b568852a2986410d96db3831426c85d8ed63dc676150d3a7db9b4926d61722 WatchSource:0}: Error finding container f0b568852a2986410d96db3831426c85d8ed63dc676150d3a7db9b4926d61722: Status 404 returned error can't find the container with id f0b568852a2986410d96db3831426c85d8ed63dc676150d3a7db9b4926d61722 Sep 12 02:17:19.088133 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:19.088044 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-848a2c-5dd6c745f9-62t4h" event={"ID":"25c8342a-cc78-4226-84c1-1c3e4a2bc018","Type":"ContainerStarted","Data":"a3806de54607e09d9afcd6a22a0307f0c8b73abf5176fd6d42d0825659f9313d"} Sep 12 02:17:19.088133 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:19.088096 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-848a2c-5dd6c745f9-62t4h" event={"ID":"25c8342a-cc78-4226-84c1-1c3e4a2bc018","Type":"ContainerStarted","Data":"f0b568852a2986410d96db3831426c85d8ed63dc676150d3a7db9b4926d61722"} Sep 12 02:17:19.088648 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:19.088169 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-848a2c-5dd6c745f9-62t4h" Sep 12 02:17:19.089892 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:19.089869 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-848a2c-55bfd6d9c4-6gkjr" event={"ID":"a0906a1c-e691-4cc0-a14a-f9fafb3b6138","Type":"ContainerStarted","Data":"d2fbb0d790de2cda6c53d330311ae83775c7a6c32d2881873142a10641a0ebe2"} Sep 12 02:17:19.090033 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:19.089904 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-848a2c-55bfd6d9c4-6gkjr" event={"ID":"a0906a1c-e691-4cc0-a14a-f9fafb3b6138","Type":"ContainerStarted","Data":"8f37ce9b2bb514d14732a0a3842d52de896216035cb9bff152fe2a2eabee5f41"} Sep 12 02:17:19.090033 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:19.089947 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-848a2c-55bfd6d9c4-6gkjr" Sep 12 02:17:19.091214 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:19.091184 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-848a2c-56b778bdc6-d2mws" event={"ID":"7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b","Type":"ContainerStarted","Data":"b62caa58e312234e47698b80a9e1042018fd500e72ae9ada38cadee57dccdfb6"} Sep 12 02:17:19.104948 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:19.104907 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-848a2c-5dd6c745f9-62t4h" podStartSLOduration=1.104889359 podStartE2EDuration="1.104889359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:17:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:17:19.103330571 +0000 UTC m=+1941.895324367" watchObservedRunningTime="2026-09-12 02:17:19.104889359 +0000 UTC m=+1941.896883159" Sep 12 02:17:19.119883 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:19.119845 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-848a2c-55bfd6d9c4-6gkjr" podStartSLOduration=1.119832548 podStartE2EDuration="1.119832548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:17:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:17:19.117768708 +0000 UTC m=+1941.909762504" watchObservedRunningTime="2026-09-12 02:17:19.119832548 +0000 UTC m=+1941.911826388" Sep 12 02:17:20.098866 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:20.098829 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-848a2c-56b778bdc6-d2mws" event={"ID":"7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b","Type":"ContainerStarted","Data":"25f49c76899ef7b6e6dc5042f0474bc1d2c94dd35161726d457710ce343ba070"} Sep 12 02:17:20.099374 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:20.099139 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-848a2c-56b778bdc6-d2mws" Sep 12 02:17:20.116369 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:20.116313 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-848a2c-56b778bdc6-d2mws" podStartSLOduration=2.663043847 podStartE2EDuration="3.116296192s" totalImagesPullingTime="453.252345ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:17:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:17:20.114526459 +0000 UTC m=+1942.906520254" watchObservedRunningTime="2026-09-12 02:17:20.116296192 +0000 UTC m=+1942.908289988" Sep 12 02:17:25.235160 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:25.235119 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:17:26.113314 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:26.113276 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-848a2c-56b778bdc6-d2mws" Sep 12 02:17:29.309223 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:29.309188 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:17:30.101135 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:30.101107 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-848a2c-55bfd6d9c4-6gkjr" Sep 12 02:17:30.101300 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:30.101263 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-848a2c-5dd6c745f9-62t4h" Sep 12 02:17:39.649361 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:39.649327 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:17:39.751554 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:39.751504 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:17:40.696794 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:40.696755 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:17:41.584797 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:41.584757 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:17:42.378486 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:42.378433 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:17:43.159671 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:43.159626 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:17:43.787397 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:43.787349 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:17:44.404394 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:44.404356 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:17:45.153621 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:45.153584 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:17:45.964719 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:45.964689 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:17:46.476169 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:46.476130 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-848a2c-56b778bdc6-d2mws"] Sep 12 02:17:46.476679 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:46.476403 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-848a2c-56b778bdc6-d2mws" podUID="7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b" containerName="maas-api" containerID="cri-o://25f49c76899ef7b6e6dc5042f0474bc1d2c94dd35161726d457710ce343ba070" gracePeriod=30 Sep 12 02:17:46.508909 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:46.508879 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:17:46.530959 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:46.530931 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-848a2c-5dd6c745f9-62t4h"] Sep 12 02:17:46.531211 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:46.531167 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-848a2c-5dd6c745f9-62t4h" podUID="25c8342a-cc78-4226-84c1-1c3e4a2bc018" containerName="payload-processing" containerID="cri-o://a3806de54607e09d9afcd6a22a0307f0c8b73abf5176fd6d42d0825659f9313d" gracePeriod=30 Sep 12 02:17:46.544570 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:46.544537 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-848a2c-55bfd6d9c4-6gkjr"] Sep 12 02:17:46.545031 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:46.545001 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-848a2c-55bfd6d9c4-6gkjr" podUID="a0906a1c-e691-4cc0-a14a-f9fafb3b6138" containerName="payload-pre-processing" containerID="cri-o://d2fbb0d790de2cda6c53d330311ae83775c7a6c32d2881873142a10641a0ebe2" gracePeriod=30 Sep 12 02:17:46.575739 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:46.575705 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:17:46.660791 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:46.660759 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:17:46.761428 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:46.761314 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:17:46.815734 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:46.815708 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-848a2c-56b778bdc6-d2mws" Sep 12 02:17:46.837377 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:46.837353 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-848a2c-5dd6c745f9-62t4h" Sep 12 02:17:46.846695 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:46.846668 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-848a2c-55bfd6d9c4-6gkjr" Sep 12 02:17:46.871094 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:46.871062 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b-openshift-service-ca\") pod \"7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b\" (UID: \"7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b\") " Sep 12 02:17:46.871288 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:46.871135 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b-maas-api-tls\") pod \"7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b\" (UID: \"7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b\") " Sep 12 02:17:46.871288 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:46.871250 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j658l\" (UniqueName: \"kubernetes.io/projected/7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b-kube-api-access-j658l\") pod \"7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b\" (UID: \"7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b\") " Sep 12 02:17:46.871288 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:46.871275 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b-metrics-certs\") pod \"7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b\" (UID: \"7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b\") " Sep 12 02:17:46.871597 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:46.871570 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b" (UID: "7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 12 02:17:46.874509 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:46.874474 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b-kube-api-access-j658l" (OuterVolumeSpecName: "kube-api-access-j658l") pod "7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b" (UID: "7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b"). InnerVolumeSpecName "kube-api-access-j658l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 12 02:17:46.874509 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:46.874492 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b" (UID: "7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 12 02:17:46.874660 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:46.874392 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b" (UID: "7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 12 02:17:46.972032 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:46.971996 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nz6rg\" (UniqueName: \"kubernetes.io/projected/25c8342a-cc78-4226-84c1-1c3e4a2bc018-kube-api-access-nz6rg\") pod \"25c8342a-cc78-4226-84c1-1c3e4a2bc018\" (UID: \"25c8342a-cc78-4226-84c1-1c3e4a2bc018\") " Sep 12 02:17:46.972222 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:46.972087 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bnrss\" (UniqueName: \"kubernetes.io/projected/a0906a1c-e691-4cc0-a14a-f9fafb3b6138-kube-api-access-bnrss\") pod \"a0906a1c-e691-4cc0-a14a-f9fafb3b6138\" (UID: \"a0906a1c-e691-4cc0-a14a-f9fafb3b6138\") " Sep 12 02:17:46.972222 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:46.972107 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0906a1c-e691-4cc0-a14a-f9fafb3b6138-plugins-config-volume\") pod \"a0906a1c-e691-4cc0-a14a-f9fafb3b6138\" (UID: \"a0906a1c-e691-4cc0-a14a-f9fafb3b6138\") " Sep 12 02:17:46.972222 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:46.972169 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25c8342a-cc78-4226-84c1-1c3e4a2bc018-plugins-config-volume\") pod \"25c8342a-cc78-4226-84c1-1c3e4a2bc018\" (UID: \"25c8342a-cc78-4226-84c1-1c3e4a2bc018\") " Sep 12 02:17:46.972409 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:46.972378 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j658l\" (UniqueName: \"kubernetes.io/projected/7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b-kube-api-access-j658l\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:17:46.972409 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:46.972390 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b-metrics-certs\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:17:46.972409 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:46.972398 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b-openshift-service-ca\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:17:46.972409 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:46.972407 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b-maas-api-tls\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:17:46.972623 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:46.972549 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0906a1c-e691-4cc0-a14a-f9fafb3b6138-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a0906a1c-e691-4cc0-a14a-f9fafb3b6138" (UID: "a0906a1c-e691-4cc0-a14a-f9fafb3b6138"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 12 02:17:46.972623 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:46.972588 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25c8342a-cc78-4226-84c1-1c3e4a2bc018-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "25c8342a-cc78-4226-84c1-1c3e4a2bc018" (UID: "25c8342a-cc78-4226-84c1-1c3e4a2bc018"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 12 02:17:46.974200 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:46.974183 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25c8342a-cc78-4226-84c1-1c3e4a2bc018-kube-api-access-nz6rg" (OuterVolumeSpecName: "kube-api-access-nz6rg") pod "25c8342a-cc78-4226-84c1-1c3e4a2bc018" (UID: "25c8342a-cc78-4226-84c1-1c3e4a2bc018"). InnerVolumeSpecName "kube-api-access-nz6rg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 12 02:17:46.974287 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:46.974269 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0906a1c-e691-4cc0-a14a-f9fafb3b6138-kube-api-access-bnrss" (OuterVolumeSpecName: "kube-api-access-bnrss") pod "a0906a1c-e691-4cc0-a14a-f9fafb3b6138" (UID: "a0906a1c-e691-4cc0-a14a-f9fafb3b6138"). InnerVolumeSpecName "kube-api-access-bnrss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 12 02:17:47.067173 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:47.067071 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:17:47.073322 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:47.073294 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25c8342a-cc78-4226-84c1-1c3e4a2bc018-plugins-config-volume\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:17:47.073322 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:47.073320 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nz6rg\" (UniqueName: \"kubernetes.io/projected/25c8342a-cc78-4226-84c1-1c3e4a2bc018-kube-api-access-nz6rg\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:17:47.073437 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:47.073330 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnrss\" (UniqueName: \"kubernetes.io/projected/a0906a1c-e691-4cc0-a14a-f9fafb3b6138-kube-api-access-bnrss\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:17:47.073437 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:47.073338 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0906a1c-e691-4cc0-a14a-f9fafb3b6138-plugins-config-volume\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:17:47.172638 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:47.172583 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:17:47.242832 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:47.242790 2579 generic.go:356] "Generic (PLEG): container finished" podID="25c8342a-cc78-4226-84c1-1c3e4a2bc018" containerID="a3806de54607e09d9afcd6a22a0307f0c8b73abf5176fd6d42d0825659f9313d" exitCode=0 Sep 12 02:17:47.243017 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:47.242840 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-848a2c-5dd6c745f9-62t4h" Sep 12 02:17:47.243017 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:47.242870 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-848a2c-5dd6c745f9-62t4h" event={"ID":"25c8342a-cc78-4226-84c1-1c3e4a2bc018","Type":"ContainerDied","Data":"a3806de54607e09d9afcd6a22a0307f0c8b73abf5176fd6d42d0825659f9313d"} Sep 12 02:17:47.243017 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:47.242927 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-848a2c-5dd6c745f9-62t4h" event={"ID":"25c8342a-cc78-4226-84c1-1c3e4a2bc018","Type":"ContainerDied","Data":"f0b568852a2986410d96db3831426c85d8ed63dc676150d3a7db9b4926d61722"} Sep 12 02:17:47.243017 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:47.242957 2579 scope.go:117] "RemoveContainer" containerID="a3806de54607e09d9afcd6a22a0307f0c8b73abf5176fd6d42d0825659f9313d" Sep 12 02:17:47.244424 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:47.244402 2579 generic.go:356] "Generic (PLEG): container finished" podID="a0906a1c-e691-4cc0-a14a-f9fafb3b6138" containerID="d2fbb0d790de2cda6c53d330311ae83775c7a6c32d2881873142a10641a0ebe2" exitCode=0 Sep 12 02:17:47.244547 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:47.244445 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-848a2c-55bfd6d9c4-6gkjr" Sep 12 02:17:47.244547 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:47.244480 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-848a2c-55bfd6d9c4-6gkjr" event={"ID":"a0906a1c-e691-4cc0-a14a-f9fafb3b6138","Type":"ContainerDied","Data":"d2fbb0d790de2cda6c53d330311ae83775c7a6c32d2881873142a10641a0ebe2"} Sep 12 02:17:47.244547 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:47.244527 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-848a2c-55bfd6d9c4-6gkjr" event={"ID":"a0906a1c-e691-4cc0-a14a-f9fafb3b6138","Type":"ContainerDied","Data":"8f37ce9b2bb514d14732a0a3842d52de896216035cb9bff152fe2a2eabee5f41"} Sep 12 02:17:47.246148 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:47.246121 2579 generic.go:356] "Generic (PLEG): container finished" podID="7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b" containerID="25f49c76899ef7b6e6dc5042f0474bc1d2c94dd35161726d457710ce343ba070" exitCode=0 Sep 12 02:17:47.246290 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:47.246181 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-848a2c-56b778bdc6-d2mws" Sep 12 02:17:47.246410 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:47.246384 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-848a2c-56b778bdc6-d2mws" event={"ID":"7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b","Type":"ContainerDied","Data":"25f49c76899ef7b6e6dc5042f0474bc1d2c94dd35161726d457710ce343ba070"} Sep 12 02:17:47.246563 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:47.246467 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-848a2c-56b778bdc6-d2mws" event={"ID":"7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b","Type":"ContainerDied","Data":"b62caa58e312234e47698b80a9e1042018fd500e72ae9ada38cadee57dccdfb6"} Sep 12 02:17:47.253948 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:47.253930 2579 scope.go:117] "RemoveContainer" containerID="a3806de54607e09d9afcd6a22a0307f0c8b73abf5176fd6d42d0825659f9313d" Sep 12 02:17:47.254181 ip-10-0-133-79 kubenswrapper[2579]: E0912 02:17:47.254162 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3806de54607e09d9afcd6a22a0307f0c8b73abf5176fd6d42d0825659f9313d\": container with ID starting with a3806de54607e09d9afcd6a22a0307f0c8b73abf5176fd6d42d0825659f9313d not found: ID does not exist" containerID="a3806de54607e09d9afcd6a22a0307f0c8b73abf5176fd6d42d0825659f9313d" Sep 12 02:17:47.254252 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:47.254184 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3806de54607e09d9afcd6a22a0307f0c8b73abf5176fd6d42d0825659f9313d"} err="failed to get container status \"a3806de54607e09d9afcd6a22a0307f0c8b73abf5176fd6d42d0825659f9313d\": rpc error: code = NotFound desc = could not find container \"a3806de54607e09d9afcd6a22a0307f0c8b73abf5176fd6d42d0825659f9313d\": container with ID starting with a3806de54607e09d9afcd6a22a0307f0c8b73abf5176fd6d42d0825659f9313d not found: ID does not exist" Sep 12 02:17:47.254252 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:47.254206 2579 scope.go:117] "RemoveContainer" containerID="d2fbb0d790de2cda6c53d330311ae83775c7a6c32d2881873142a10641a0ebe2" Sep 12 02:17:47.263475 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:47.263435 2579 scope.go:117] "RemoveContainer" containerID="d2fbb0d790de2cda6c53d330311ae83775c7a6c32d2881873142a10641a0ebe2" Sep 12 02:17:47.263754 ip-10-0-133-79 kubenswrapper[2579]: E0912 02:17:47.263733 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2fbb0d790de2cda6c53d330311ae83775c7a6c32d2881873142a10641a0ebe2\": container with ID starting with d2fbb0d790de2cda6c53d330311ae83775c7a6c32d2881873142a10641a0ebe2 not found: ID does not exist" containerID="d2fbb0d790de2cda6c53d330311ae83775c7a6c32d2881873142a10641a0ebe2" Sep 12 02:17:47.263865 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:47.263762 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2fbb0d790de2cda6c53d330311ae83775c7a6c32d2881873142a10641a0ebe2"} err="failed to get container status \"d2fbb0d790de2cda6c53d330311ae83775c7a6c32d2881873142a10641a0ebe2\": rpc error: code = NotFound desc = could not find container \"d2fbb0d790de2cda6c53d330311ae83775c7a6c32d2881873142a10641a0ebe2\": container with ID starting with d2fbb0d790de2cda6c53d330311ae83775c7a6c32d2881873142a10641a0ebe2 not found: ID does not exist" Sep 12 02:17:47.263865 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:47.263781 2579 scope.go:117] "RemoveContainer" containerID="25f49c76899ef7b6e6dc5042f0474bc1d2c94dd35161726d457710ce343ba070" Sep 12 02:17:47.272378 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:47.272339 2579 scope.go:117] "RemoveContainer" containerID="25f49c76899ef7b6e6dc5042f0474bc1d2c94dd35161726d457710ce343ba070" Sep 12 02:17:47.272716 ip-10-0-133-79 kubenswrapper[2579]: E0912 02:17:47.272683 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25f49c76899ef7b6e6dc5042f0474bc1d2c94dd35161726d457710ce343ba070\": container with ID starting with 25f49c76899ef7b6e6dc5042f0474bc1d2c94dd35161726d457710ce343ba070 not found: ID does not exist" containerID="25f49c76899ef7b6e6dc5042f0474bc1d2c94dd35161726d457710ce343ba070" Sep 12 02:17:47.272817 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:47.272724 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25f49c76899ef7b6e6dc5042f0474bc1d2c94dd35161726d457710ce343ba070"} err="failed to get container status \"25f49c76899ef7b6e6dc5042f0474bc1d2c94dd35161726d457710ce343ba070\": rpc error: code = NotFound desc = could not find container \"25f49c76899ef7b6e6dc5042f0474bc1d2c94dd35161726d457710ce343ba070\": container with ID starting with 25f49c76899ef7b6e6dc5042f0474bc1d2c94dd35161726d457710ce343ba070 not found: ID does not exist" Sep 12 02:17:47.273640 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:47.273622 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-848a2c-5dd6c745f9-62t4h"] Sep 12 02:17:47.278966 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:47.278948 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-848a2c-5dd6c745f9-62t4h"] Sep 12 02:17:47.289061 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:47.289034 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-848a2c-56b778bdc6-d2mws"] Sep 12 02:17:47.294189 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:47.294166 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-848a2c-56b778bdc6-d2mws"] Sep 12 02:17:47.304511 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:47.304490 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-848a2c-55bfd6d9c4-6gkjr"] Sep 12 02:17:47.309941 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:47.309923 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-848a2c-55bfd6d9c4-6gkjr"] Sep 12 02:17:47.728820 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:47.728791 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25c8342a-cc78-4226-84c1-1c3e4a2bc018" path="/var/lib/kubelet/pods/25c8342a-cc78-4226-84c1-1c3e4a2bc018/volumes" Sep 12 02:17:47.729197 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:47.729184 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b" path="/var/lib/kubelet/pods/7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b/volumes" Sep 12 02:17:47.729515 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:47.729503 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0906a1c-e691-4cc0-a14a-f9fafb3b6138" path="/var/lib/kubelet/pods/a0906a1c-e691-4cc0-a14a-f9fafb3b6138/volumes" Sep 12 02:17:47.852201 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:47.852101 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:17:48.482889 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:48.482852 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:17:49.242910 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:49.242864 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:17:49.976930 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:49.976895 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:17:51.023473 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:51.023421 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:17:52.850256 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:52.850208 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:17:52.951011 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:52.950957 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:17:53.118563 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:53.118473 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:17:57.081083 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:57.081047 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:17:57.827708 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:57.827669 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:17:58.572860 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:58.572827 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:17:59.224438 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:59.224397 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:17:59.967917 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:17:59.967871 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:18:00.760242 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:18:00.760211 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:18:09.822522 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:18:09.822483 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:18:12.378364 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:18:12.378326 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:18:12.446266 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:18:12.446236 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:18:12.509788 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:18:12.509750 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:18:12.568685 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:18:12.568654 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:18:12.953078 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:18:12.953043 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:18:13.070309 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:18:13.070278 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:18:13.878642 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:18:13.878612 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:18:14.579834 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:18:14.579797 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:18:15.246995 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:18:15.246962 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:18:17.982499 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:18:17.982439 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:18:23.325872 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:18:23.325839 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:18:38.229951 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:18:38.229916 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:18:41.357258 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:18:41.357213 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:18:54.075672 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:18:54.075640 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:18:54.920122 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:18:54.920090 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:19:00.549812 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:00.549775 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:19:14.474168 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:14.474112 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:19:14.534511 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:14.534474 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:19:14.798704 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:14.798624 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:19:15.260342 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:15.260306 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:19:16.154062 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:16.154023 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:19:16.869494 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:16.869431 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:19:17.600648 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:17.600613 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:19:18.367143 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:18.367075 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:19:25.572858 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:25.572767 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-01570e-59f8689478-tfnbh"] Sep 12 02:19:25.573367 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:25.573035 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01570e-59f8689478-tfnbh" podUID="d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1" containerName="maas-api" containerID="cri-o://adbd597855df1134d878204f7936688a16f4ed3043c21de3d51073f0668f8b67" gracePeriod=30 Sep 12 02:19:25.603916 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:25.603876 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:19:25.635525 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:25.635496 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-01570e-665c84644b-f9zxk"] Sep 12 02:19:25.635745 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:25.635719 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01570e-665c84644b-f9zxk" podUID="719dd67d-9839-4be1-99fa-c51f22340e31" containerName="payload-pre-processing" containerID="cri-o://021985d21265ca2cc1c811d0528e5950a85201f571d5c965633a218e3181a95f" gracePeriod=30 Sep 12 02:19:25.702567 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:25.702531 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:19:25.758109 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:25.758079 2579 generic.go:356] "Generic (PLEG): container finished" podID="d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1" containerID="adbd597855df1134d878204f7936688a16f4ed3043c21de3d51073f0668f8b67" exitCode=0 Sep 12 02:19:25.758279 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:25.758157 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01570e-59f8689478-tfnbh" event={"ID":"d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1","Type":"ContainerDied","Data":"adbd597855df1134d878204f7936688a16f4ed3043c21de3d51073f0668f8b67"} Sep 12 02:19:25.760052 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:25.760025 2579 generic.go:356] "Generic (PLEG): container finished" podID="719dd67d-9839-4be1-99fa-c51f22340e31" containerID="021985d21265ca2cc1c811d0528e5950a85201f571d5c965633a218e3181a95f" exitCode=0 Sep 12 02:19:25.760177 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:25.760092 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01570e-665c84644b-f9zxk" event={"ID":"719dd67d-9839-4be1-99fa-c51f22340e31","Type":"ContainerDied","Data":"021985d21265ca2cc1c811d0528e5950a85201f571d5c965633a218e3181a95f"} Sep 12 02:19:25.803246 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:25.802895 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:19:25.863667 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:25.863623 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:19:25.896368 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:25.896341 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01570e-59f8689478-tfnbh" Sep 12 02:19:25.923660 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:25.923634 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01570e-665c84644b-f9zxk" Sep 12 02:19:25.947906 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:25.947882 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1-openshift-service-ca\") pod \"d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1\" (UID: \"d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1\") " Sep 12 02:19:25.948063 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:25.947915 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9fbgf\" (UniqueName: \"kubernetes.io/projected/719dd67d-9839-4be1-99fa-c51f22340e31-kube-api-access-9fbgf\") pod \"719dd67d-9839-4be1-99fa-c51f22340e31\" (UID: \"719dd67d-9839-4be1-99fa-c51f22340e31\") " Sep 12 02:19:25.948063 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:25.947950 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tgp28\" (UniqueName: \"kubernetes.io/projected/d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1-kube-api-access-tgp28\") pod \"d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1\" (UID: \"d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1\") " Sep 12 02:19:25.948063 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:25.947972 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/719dd67d-9839-4be1-99fa-c51f22340e31-plugins-config-volume\") pod \"719dd67d-9839-4be1-99fa-c51f22340e31\" (UID: \"719dd67d-9839-4be1-99fa-c51f22340e31\") " Sep 12 02:19:25.948063 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:25.948039 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1-metrics-certs\") pod \"d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1\" (UID: \"d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1\") " Sep 12 02:19:25.948281 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:25.948150 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1-maas-api-tls\") pod \"d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1\" (UID: \"d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1\") " Sep 12 02:19:25.948379 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:25.948352 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1" (UID: "d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 12 02:19:25.948483 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:25.948384 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/719dd67d-9839-4be1-99fa-c51f22340e31-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "719dd67d-9839-4be1-99fa-c51f22340e31" (UID: "719dd67d-9839-4be1-99fa-c51f22340e31"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 12 02:19:25.948545 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:25.948532 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1-openshift-service-ca\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:19:25.948600 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:25.948551 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/719dd67d-9839-4be1-99fa-c51f22340e31-plugins-config-volume\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:19:25.950273 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:25.950244 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/719dd67d-9839-4be1-99fa-c51f22340e31-kube-api-access-9fbgf" (OuterVolumeSpecName: "kube-api-access-9fbgf") pod "719dd67d-9839-4be1-99fa-c51f22340e31" (UID: "719dd67d-9839-4be1-99fa-c51f22340e31"). InnerVolumeSpecName "kube-api-access-9fbgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 12 02:19:25.950442 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:25.950425 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1" (UID: "d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 12 02:19:25.950839 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:25.950819 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1" (UID: "d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 12 02:19:25.950929 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:25.950844 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1-kube-api-access-tgp28" (OuterVolumeSpecName: "kube-api-access-tgp28") pod "d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1" (UID: "d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1"). InnerVolumeSpecName "kube-api-access-tgp28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 12 02:19:26.049697 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:26.049664 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fbgf\" (UniqueName: \"kubernetes.io/projected/719dd67d-9839-4be1-99fa-c51f22340e31-kube-api-access-9fbgf\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:19:26.049697 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:26.049692 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgp28\" (UniqueName: \"kubernetes.io/projected/d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1-kube-api-access-tgp28\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:19:26.049697 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:26.049703 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1-metrics-certs\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:19:26.049922 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:26.049711 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1-maas-api-tls\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:19:26.242463 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:26.242424 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:19:26.343328 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:26.343290 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:19:26.767934 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:26.767904 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01570e-59f8689478-tfnbh" Sep 12 02:19:26.767934 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:26.767907 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01570e-59f8689478-tfnbh" event={"ID":"d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1","Type":"ContainerDied","Data":"8aa353febdd27de324c5320b42b50770dec7dcaca906e9d474d3c335b7d7cd0d"} Sep 12 02:19:26.768428 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:26.767979 2579 scope.go:117] "RemoveContainer" containerID="adbd597855df1134d878204f7936688a16f4ed3043c21de3d51073f0668f8b67" Sep 12 02:19:26.769540 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:26.769522 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01570e-665c84644b-f9zxk" Sep 12 02:19:26.769672 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:26.769647 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01570e-665c84644b-f9zxk" event={"ID":"719dd67d-9839-4be1-99fa-c51f22340e31","Type":"ContainerDied","Data":"cab0876ef59d0ca5aa56a5d93d6448bcc4ca22c251eb9f0b33f3af9e19980063"} Sep 12 02:19:26.777120 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:26.777104 2579 scope.go:117] "RemoveContainer" containerID="021985d21265ca2cc1c811d0528e5950a85201f571d5c965633a218e3181a95f" Sep 12 02:19:26.791787 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:26.791768 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-01570e-665c84644b-f9zxk"] Sep 12 02:19:26.795901 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:26.795879 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-01570e-665c84644b-f9zxk"] Sep 12 02:19:26.809782 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:26.809762 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-01570e-59f8689478-tfnbh"] Sep 12 02:19:26.820289 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:26.820266 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-01570e-59f8689478-tfnbh"] Sep 12 02:19:27.034838 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:27.034805 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:19:27.726633 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:27.726598 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="719dd67d-9839-4be1-99fa-c51f22340e31" path="/var/lib/kubelet/pods/719dd67d-9839-4be1-99fa-c51f22340e31/volumes" Sep 12 02:19:27.726999 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:27.726986 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1" path="/var/lib/kubelet/pods/d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1/volumes" Sep 12 02:19:28.008529 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:28.008427 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:19:28.920440 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:28.920408 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:19:31.933517 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:31.933442 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:19:31.988562 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:31.988525 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:19:32.224391 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:32.224295 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:19:32.732648 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:32.732612 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:19:33.560169 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:33.560129 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:19:34.189944 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:34.189895 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:19:34.957249 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:34.957213 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:19:42.986193 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:42.986158 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a744d5-69c5c5bd6c-fnbt9"] Sep 12 02:19:42.986699 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:42.986412 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a744d5-69c5c5bd6c-fnbt9" podUID="4b6147b2-1b11-42b3-a824-fe7525170f7d" containerName="maas-api" containerID="cri-o://9df793718fe1221f483d55014956d02c07bd8bc159795d1c75fa1da2a246520a" gracePeriod=30 Sep 12 02:19:43.016468 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:43.016426 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:19:43.115973 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:43.115932 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:19:43.189586 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:43.189557 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:19:43.232233 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:43.232205 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a744d5-69c5c5bd6c-fnbt9" Sep 12 02:19:43.252578 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:43.252489 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:19:43.312127 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:43.312094 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b6147b2-1b11-42b3-a824-fe7525170f7d-maas-api-tls\") pod \"4b6147b2-1b11-42b3-a824-fe7525170f7d\" (UID: \"4b6147b2-1b11-42b3-a824-fe7525170f7d\") " Sep 12 02:19:43.312335 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:43.312162 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7xpdz\" (UniqueName: \"kubernetes.io/projected/4b6147b2-1b11-42b3-a824-fe7525170f7d-kube-api-access-7xpdz\") pod \"4b6147b2-1b11-42b3-a824-fe7525170f7d\" (UID: \"4b6147b2-1b11-42b3-a824-fe7525170f7d\") " Sep 12 02:19:43.312335 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:43.312297 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b6147b2-1b11-42b3-a824-fe7525170f7d-openshift-service-ca\") pod \"4b6147b2-1b11-42b3-a824-fe7525170f7d\" (UID: \"4b6147b2-1b11-42b3-a824-fe7525170f7d\") " Sep 12 02:19:43.312418 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:43.312403 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b6147b2-1b11-42b3-a824-fe7525170f7d-metrics-certs\") pod \"4b6147b2-1b11-42b3-a824-fe7525170f7d\" (UID: \"4b6147b2-1b11-42b3-a824-fe7525170f7d\") " Sep 12 02:19:43.312716 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:43.312682 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b6147b2-1b11-42b3-a824-fe7525170f7d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4b6147b2-1b11-42b3-a824-fe7525170f7d" (UID: "4b6147b2-1b11-42b3-a824-fe7525170f7d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 12 02:19:43.312826 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:43.312788 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b6147b2-1b11-42b3-a824-fe7525170f7d-openshift-service-ca\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:19:43.314826 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:43.314802 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b6147b2-1b11-42b3-a824-fe7525170f7d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4b6147b2-1b11-42b3-a824-fe7525170f7d" (UID: "4b6147b2-1b11-42b3-a824-fe7525170f7d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 12 02:19:43.314980 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:43.314954 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b6147b2-1b11-42b3-a824-fe7525170f7d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4b6147b2-1b11-42b3-a824-fe7525170f7d" (UID: "4b6147b2-1b11-42b3-a824-fe7525170f7d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 12 02:19:43.315055 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:43.314956 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b6147b2-1b11-42b3-a824-fe7525170f7d-kube-api-access-7xpdz" (OuterVolumeSpecName: "kube-api-access-7xpdz") pod "4b6147b2-1b11-42b3-a824-fe7525170f7d" (UID: "4b6147b2-1b11-42b3-a824-fe7525170f7d"). InnerVolumeSpecName "kube-api-access-7xpdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 12 02:19:43.413905 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:43.413855 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b6147b2-1b11-42b3-a824-fe7525170f7d-maas-api-tls\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:19:43.413905 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:43.413899 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xpdz\" (UniqueName: \"kubernetes.io/projected/4b6147b2-1b11-42b3-a824-fe7525170f7d-kube-api-access-7xpdz\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:19:43.413905 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:43.413910 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b6147b2-1b11-42b3-a824-fe7525170f7d-metrics-certs\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:19:43.648554 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:43.648522 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:19:43.741954 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:43.741928 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:19:43.862679 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:43.862651 2579 generic.go:356] "Generic (PLEG): container finished" podID="4b6147b2-1b11-42b3-a824-fe7525170f7d" containerID="9df793718fe1221f483d55014956d02c07bd8bc159795d1c75fa1da2a246520a" exitCode=0 Sep 12 02:19:43.862846 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:43.862692 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a744d5-69c5c5bd6c-fnbt9" event={"ID":"4b6147b2-1b11-42b3-a824-fe7525170f7d","Type":"ContainerDied","Data":"9df793718fe1221f483d55014956d02c07bd8bc159795d1c75fa1da2a246520a"} Sep 12 02:19:43.862846 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:43.862701 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a744d5-69c5c5bd6c-fnbt9" Sep 12 02:19:43.862846 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:43.862736 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a744d5-69c5c5bd6c-fnbt9" event={"ID":"4b6147b2-1b11-42b3-a824-fe7525170f7d","Type":"ContainerDied","Data":"c888817c0f2f9632e873c8cb52defa1cac0fa7a201e3983c8271d9d0638351bb"} Sep 12 02:19:43.862846 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:43.862756 2579 scope.go:117] "RemoveContainer" containerID="9df793718fe1221f483d55014956d02c07bd8bc159795d1c75fa1da2a246520a" Sep 12 02:19:43.880389 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:43.880343 2579 scope.go:117] "RemoveContainer" containerID="9df793718fe1221f483d55014956d02c07bd8bc159795d1c75fa1da2a246520a" Sep 12 02:19:43.880912 ip-10-0-133-79 kubenswrapper[2579]: E0912 02:19:43.880879 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9df793718fe1221f483d55014956d02c07bd8bc159795d1c75fa1da2a246520a\": container with ID starting with 9df793718fe1221f483d55014956d02c07bd8bc159795d1c75fa1da2a246520a not found: ID does not exist" containerID="9df793718fe1221f483d55014956d02c07bd8bc159795d1c75fa1da2a246520a" Sep 12 02:19:43.881030 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:43.880921 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9df793718fe1221f483d55014956d02c07bd8bc159795d1c75fa1da2a246520a"} err="failed to get container status \"9df793718fe1221f483d55014956d02c07bd8bc159795d1c75fa1da2a246520a\": rpc error: code = NotFound desc = could not find container \"9df793718fe1221f483d55014956d02c07bd8bc159795d1c75fa1da2a246520a\": container with ID starting with 9df793718fe1221f483d55014956d02c07bd8bc159795d1c75fa1da2a246520a not found: ID does not exist" Sep 12 02:19:43.882915 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:43.882894 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a744d5-69c5c5bd6c-fnbt9"] Sep 12 02:19:43.889220 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:43.889197 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a744d5-69c5c5bd6c-fnbt9"] Sep 12 02:19:44.392208 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:44.392152 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:19:45.727204 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:45.727173 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b6147b2-1b11-42b3-a824-fe7525170f7d" path="/var/lib/kubelet/pods/4b6147b2-1b11-42b3-a824-fe7525170f7d/volumes" Sep 12 02:19:45.921418 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:45.921379 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:19:49.561299 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:49.561268 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:19:49.622142 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:49.622103 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:19:49.832070 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:49.831979 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:19:50.461660 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:50.461620 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:19:51.171001 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:51.170958 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:19:51.776966 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:19:51.776926 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:20:00.677390 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:20:00.677358 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fbb6d9-7cb9c56db4-h2gdc"] Sep 12 02:20:00.677905 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:20:00.677623 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fbb6d9-7cb9c56db4-h2gdc" podUID="8cc047c2-a082-41ff-849d-391241409035" containerName="maas-api" containerID="cri-o://f26ffe20e267ea71c2c6e3fb5d932c59da1ada488d5503e3deaaa90f98fb2b3e" gracePeriod=30 Sep 12 02:20:00.704943 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:20:00.704913 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:20:00.804966 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:20:00.804929 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:20:00.881386 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:20:00.881348 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:20:00.923787 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:20:00.923765 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fbb6d9-7cb9c56db4-h2gdc" Sep 12 02:20:00.945153 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:20:00.944739 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:20:00.948832 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:20:00.948806 2579 generic.go:356] "Generic (PLEG): container finished" podID="8cc047c2-a082-41ff-849d-391241409035" containerID="f26ffe20e267ea71c2c6e3fb5d932c59da1ada488d5503e3deaaa90f98fb2b3e" exitCode=0 Sep 12 02:20:00.948966 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:20:00.948845 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fbb6d9-7cb9c56db4-h2gdc" Sep 12 02:20:00.948966 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:20:00.948854 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fbb6d9-7cb9c56db4-h2gdc" event={"ID":"8cc047c2-a082-41ff-849d-391241409035","Type":"ContainerDied","Data":"f26ffe20e267ea71c2c6e3fb5d932c59da1ada488d5503e3deaaa90f98fb2b3e"} Sep 12 02:20:00.948966 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:20:00.948908 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fbb6d9-7cb9c56db4-h2gdc" event={"ID":"8cc047c2-a082-41ff-849d-391241409035","Type":"ContainerDied","Data":"eff42daeb83dcb0f98a97dd85a7ebdbd01a47dfedca5a941316ff2ac99a65ea2"} Sep 12 02:20:00.948966 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:20:00.948934 2579 scope.go:117] "RemoveContainer" containerID="f26ffe20e267ea71c2c6e3fb5d932c59da1ada488d5503e3deaaa90f98fb2b3e" Sep 12 02:20:00.959093 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:20:00.959069 2579 scope.go:117] "RemoveContainer" containerID="f26ffe20e267ea71c2c6e3fb5d932c59da1ada488d5503e3deaaa90f98fb2b3e" Sep 12 02:20:00.959361 ip-10-0-133-79 kubenswrapper[2579]: E0912 02:20:00.959342 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f26ffe20e267ea71c2c6e3fb5d932c59da1ada488d5503e3deaaa90f98fb2b3e\": container with ID starting with f26ffe20e267ea71c2c6e3fb5d932c59da1ada488d5503e3deaaa90f98fb2b3e not found: ID does not exist" containerID="f26ffe20e267ea71c2c6e3fb5d932c59da1ada488d5503e3deaaa90f98fb2b3e" Sep 12 02:20:00.959439 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:20:00.959372 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f26ffe20e267ea71c2c6e3fb5d932c59da1ada488d5503e3deaaa90f98fb2b3e"} err="failed to get container status \"f26ffe20e267ea71c2c6e3fb5d932c59da1ada488d5503e3deaaa90f98fb2b3e\": rpc error: code = NotFound desc = could not find container \"f26ffe20e267ea71c2c6e3fb5d932c59da1ada488d5503e3deaaa90f98fb2b3e\": container with ID starting with f26ffe20e267ea71c2c6e3fb5d932c59da1ada488d5503e3deaaa90f98fb2b3e not found: ID does not exist" Sep 12 02:20:01.089320 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:20:01.089271 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cc047c2-a082-41ff-849d-391241409035-metrics-certs\") pod \"8cc047c2-a082-41ff-849d-391241409035\" (UID: \"8cc047c2-a082-41ff-849d-391241409035\") " Sep 12 02:20:01.089512 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:20:01.089394 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cc047c2-a082-41ff-849d-391241409035-maas-api-tls\") pod \"8cc047c2-a082-41ff-849d-391241409035\" (UID: \"8cc047c2-a082-41ff-849d-391241409035\") " Sep 12 02:20:01.089512 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:20:01.089479 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cc047c2-a082-41ff-849d-391241409035-openshift-service-ca\") pod \"8cc047c2-a082-41ff-849d-391241409035\" (UID: \"8cc047c2-a082-41ff-849d-391241409035\") " Sep 12 02:20:01.089658 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:20:01.089517 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g5qwc\" (UniqueName: \"kubernetes.io/projected/8cc047c2-a082-41ff-849d-391241409035-kube-api-access-g5qwc\") pod \"8cc047c2-a082-41ff-849d-391241409035\" (UID: \"8cc047c2-a082-41ff-849d-391241409035\") " Sep 12 02:20:01.089859 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:20:01.089827 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cc047c2-a082-41ff-849d-391241409035-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8cc047c2-a082-41ff-849d-391241409035" (UID: "8cc047c2-a082-41ff-849d-391241409035"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 12 02:20:01.091650 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:20:01.091631 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cc047c2-a082-41ff-849d-391241409035-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8cc047c2-a082-41ff-849d-391241409035" (UID: "8cc047c2-a082-41ff-849d-391241409035"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 12 02:20:01.091769 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:20:01.091738 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cc047c2-a082-41ff-849d-391241409035-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8cc047c2-a082-41ff-849d-391241409035" (UID: "8cc047c2-a082-41ff-849d-391241409035"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 12 02:20:01.091814 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:20:01.091768 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cc047c2-a082-41ff-849d-391241409035-kube-api-access-g5qwc" (OuterVolumeSpecName: "kube-api-access-g5qwc") pod "8cc047c2-a082-41ff-849d-391241409035" (UID: "8cc047c2-a082-41ff-849d-391241409035"). InnerVolumeSpecName "kube-api-access-g5qwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 12 02:20:01.190235 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:20:01.190205 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cc047c2-a082-41ff-849d-391241409035-maas-api-tls\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:20:01.190235 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:20:01.190230 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cc047c2-a082-41ff-849d-391241409035-openshift-service-ca\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:20:01.190235 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:20:01.190240 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5qwc\" (UniqueName: \"kubernetes.io/projected/8cc047c2-a082-41ff-849d-391241409035-kube-api-access-g5qwc\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:20:01.190416 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:20:01.190249 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cc047c2-a082-41ff-849d-391241409035-metrics-certs\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:20:01.270830 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:20:01.270803 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fbb6d9-7cb9c56db4-h2gdc"] Sep 12 02:20:01.272555 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:20:01.272531 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fbb6d9-7cb9c56db4-h2gdc"] Sep 12 02:20:01.280124 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:20:01.280098 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:20:01.454523 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:20:01.454483 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:20:01.725505 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:20:01.725476 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cc047c2-a082-41ff-849d-391241409035" path="/var/lib/kubelet/pods/8cc047c2-a082-41ff-849d-391241409035/volumes" Sep 12 02:20:03.326328 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:20:03.326289 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:20:07.150688 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:20:07.150653 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:20:07.253757 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:20:07.253728 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:20:07.443704 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:20:07.443603 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:20:11.717251 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:20:11.717208 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:20:12.421298 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:20:12.421243 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:20:40.095943 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:20:40.095909 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:20:46.801404 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:20:46.801367 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:20:46.854279 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:20:46.854240 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:21:00.926001 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:21:00.925958 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:21:33.156322 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:21:33.156285 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:21:33.728532 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:21:33.728499 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:21:34.936504 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:21:34.936470 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:22:00.030378 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:22:00.030339 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:22:07.926463 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:22:07.926426 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:22:14.816120 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:22:14.816084 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:22:22.123567 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:22:22.123486 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:22:37.126891 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:22:37.126853 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:22:44.830220 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:22:44.830184 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:24:14.131169 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:24:14.131138 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:24:14.717208 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:24:14.717174 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:24:21.521725 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:24:21.521694 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:24:31.023937 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:24:31.023903 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:24:42.767710 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:24:42.767672 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:24:45.218417 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:24:45.218381 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:24:49.361717 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:24:49.361685 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:24:52.343471 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:24:52.343421 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:24:55.231050 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:24:55.231013 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:25:58.831159 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:25:58.831124 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:26:00.346470 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:00.346430 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:26:01.017739 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:01.017702 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:26:08.631032 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:08.630998 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:26:16.729243 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:16.729205 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:26:16.733410 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:16.733383 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-77b9747d97-tmqrt"] Sep 12 02:26:16.733844 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:16.733831 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0906a1c-e691-4cc0-a14a-f9fafb3b6138" containerName="payload-pre-processing" Sep 12 02:26:16.733896 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:16.733847 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0906a1c-e691-4cc0-a14a-f9fafb3b6138" containerName="payload-pre-processing" Sep 12 02:26:16.733896 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:16.733857 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b6147b2-1b11-42b3-a824-fe7525170f7d" containerName="maas-api" Sep 12 02:26:16.733896 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:16.733865 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b6147b2-1b11-42b3-a824-fe7525170f7d" containerName="maas-api" Sep 12 02:26:16.733896 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:16.733881 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="719dd67d-9839-4be1-99fa-c51f22340e31" containerName="payload-pre-processing" Sep 12 02:26:16.733896 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:16.733886 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="719dd67d-9839-4be1-99fa-c51f22340e31" containerName="payload-pre-processing" Sep 12 02:26:16.733896 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:16.733898 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b" containerName="maas-api" Sep 12 02:26:16.734144 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:16.733903 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b" containerName="maas-api" Sep 12 02:26:16.734144 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:16.733910 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1" containerName="maas-api" Sep 12 02:26:16.734144 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:16.733915 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1" containerName="maas-api" Sep 12 02:26:16.734144 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:16.733927 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cc047c2-a082-41ff-849d-391241409035" containerName="maas-api" Sep 12 02:26:16.734144 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:16.733932 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cc047c2-a082-41ff-849d-391241409035" containerName="maas-api" Sep 12 02:26:16.734144 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:16.733941 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25c8342a-cc78-4226-84c1-1c3e4a2bc018" containerName="payload-processing" Sep 12 02:26:16.734144 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:16.733949 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="25c8342a-cc78-4226-84c1-1c3e4a2bc018" containerName="payload-processing" Sep 12 02:26:16.734144 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:16.734014 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0906a1c-e691-4cc0-a14a-f9fafb3b6138" containerName="payload-pre-processing" Sep 12 02:26:16.734144 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:16.734023 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="719dd67d-9839-4be1-99fa-c51f22340e31" containerName="payload-pre-processing" Sep 12 02:26:16.734144 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:16.734032 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="25c8342a-cc78-4226-84c1-1c3e4a2bc018" containerName="payload-processing" Sep 12 02:26:16.734144 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:16.734043 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b6147b2-1b11-42b3-a824-fe7525170f7d" containerName="maas-api" Sep 12 02:26:16.734144 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:16.734054 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c1f5068-b488-4fc4-bac0-bb9c0cb41a9b" containerName="maas-api" Sep 12 02:26:16.734144 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:16.734059 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2f52ed2-2d9b-40a0-88d7-0fe94aacdcd1" containerName="maas-api" Sep 12 02:26:16.734144 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:16.734065 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cc047c2-a082-41ff-849d-391241409035" containerName="maas-api" Sep 12 02:26:16.737285 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:16.737270 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-77b9747d97-tmqrt" Sep 12 02:26:16.740545 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:16.740521 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 12 02:26:16.740673 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:16.740551 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 12 02:26:16.740673 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:16.740571 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-565nn\"" Sep 12 02:26:16.740673 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:16.740526 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 12 02:26:16.740673 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:16.740524 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 12 02:26:16.745391 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:16.745361 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-77b9747d97-tmqrt"] Sep 12 02:26:16.834795 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:16.834760 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf2254fd-d416-41c3-a7d5-83ba747117f9-metrics-certs\") pod \"maas-api-77b9747d97-tmqrt\" (UID: \"bf2254fd-d416-41c3-a7d5-83ba747117f9\") " pod="odh-ai-gateway-infra/maas-api-77b9747d97-tmqrt" Sep 12 02:26:16.834795 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:16.834804 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf2254fd-d416-41c3-a7d5-83ba747117f9-openshift-service-ca\") pod \"maas-api-77b9747d97-tmqrt\" (UID: \"bf2254fd-d416-41c3-a7d5-83ba747117f9\") " pod="odh-ai-gateway-infra/maas-api-77b9747d97-tmqrt" Sep 12 02:26:16.835031 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:16.834860 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqlv5\" (UniqueName: \"kubernetes.io/projected/bf2254fd-d416-41c3-a7d5-83ba747117f9-kube-api-access-qqlv5\") pod \"maas-api-77b9747d97-tmqrt\" (UID: \"bf2254fd-d416-41c3-a7d5-83ba747117f9\") " pod="odh-ai-gateway-infra/maas-api-77b9747d97-tmqrt" Sep 12 02:26:16.835031 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:16.834945 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf2254fd-d416-41c3-a7d5-83ba747117f9-maas-api-tls\") pod \"maas-api-77b9747d97-tmqrt\" (UID: \"bf2254fd-d416-41c3-a7d5-83ba747117f9\") " pod="odh-ai-gateway-infra/maas-api-77b9747d97-tmqrt" Sep 12 02:26:16.936294 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:16.936254 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf2254fd-d416-41c3-a7d5-83ba747117f9-maas-api-tls\") pod \"maas-api-77b9747d97-tmqrt\" (UID: \"bf2254fd-d416-41c3-a7d5-83ba747117f9\") " pod="odh-ai-gateway-infra/maas-api-77b9747d97-tmqrt" Sep 12 02:26:16.936520 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:16.936316 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf2254fd-d416-41c3-a7d5-83ba747117f9-metrics-certs\") pod \"maas-api-77b9747d97-tmqrt\" (UID: \"bf2254fd-d416-41c3-a7d5-83ba747117f9\") " pod="odh-ai-gateway-infra/maas-api-77b9747d97-tmqrt" Sep 12 02:26:16.936520 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:16.936340 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf2254fd-d416-41c3-a7d5-83ba747117f9-openshift-service-ca\") pod \"maas-api-77b9747d97-tmqrt\" (UID: \"bf2254fd-d416-41c3-a7d5-83ba747117f9\") " pod="odh-ai-gateway-infra/maas-api-77b9747d97-tmqrt" Sep 12 02:26:16.936520 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:16.936371 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqlv5\" (UniqueName: \"kubernetes.io/projected/bf2254fd-d416-41c3-a7d5-83ba747117f9-kube-api-access-qqlv5\") pod \"maas-api-77b9747d97-tmqrt\" (UID: \"bf2254fd-d416-41c3-a7d5-83ba747117f9\") " pod="odh-ai-gateway-infra/maas-api-77b9747d97-tmqrt" Sep 12 02:26:16.937049 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:16.937029 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf2254fd-d416-41c3-a7d5-83ba747117f9-openshift-service-ca\") pod \"maas-api-77b9747d97-tmqrt\" (UID: \"bf2254fd-d416-41c3-a7d5-83ba747117f9\") " pod="odh-ai-gateway-infra/maas-api-77b9747d97-tmqrt" Sep 12 02:26:16.938857 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:16.938836 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf2254fd-d416-41c3-a7d5-83ba747117f9-maas-api-tls\") pod \"maas-api-77b9747d97-tmqrt\" (UID: \"bf2254fd-d416-41c3-a7d5-83ba747117f9\") " pod="odh-ai-gateway-infra/maas-api-77b9747d97-tmqrt" Sep 12 02:26:16.938951 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:16.938884 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf2254fd-d416-41c3-a7d5-83ba747117f9-metrics-certs\") pod \"maas-api-77b9747d97-tmqrt\" (UID: \"bf2254fd-d416-41c3-a7d5-83ba747117f9\") " pod="odh-ai-gateway-infra/maas-api-77b9747d97-tmqrt" Sep 12 02:26:16.943190 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:16.943171 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqlv5\" (UniqueName: \"kubernetes.io/projected/bf2254fd-d416-41c3-a7d5-83ba747117f9-kube-api-access-qqlv5\") pod \"maas-api-77b9747d97-tmqrt\" (UID: \"bf2254fd-d416-41c3-a7d5-83ba747117f9\") " pod="odh-ai-gateway-infra/maas-api-77b9747d97-tmqrt" Sep 12 02:26:17.049947 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:17.049848 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-77b9747d97-tmqrt" Sep 12 02:26:17.322209 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:17.322182 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-77b9747d97-tmqrt"] Sep 12 02:26:17.324469 ip-10-0-133-79 kubenswrapper[2579]: W0912 02:26:17.324423 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf2254fd_d416_41c3_a7d5_83ba747117f9.slice/crio-3ab17178c1050333cb645323b62580237133cc6036ef0d689381060b69396029 WatchSource:0}: Error finding container 3ab17178c1050333cb645323b62580237133cc6036ef0d689381060b69396029: Status 404 returned error can't find the container with id 3ab17178c1050333cb645323b62580237133cc6036ef0d689381060b69396029 Sep 12 02:26:17.325816 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:17.325798 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 12 02:26:17.794136 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:17.794108 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-77b9747d97-tmqrt" event={"ID":"bf2254fd-d416-41c3-a7d5-83ba747117f9","Type":"ContainerStarted","Data":"3ab17178c1050333cb645323b62580237133cc6036ef0d689381060b69396029"} Sep 12 02:26:18.800655 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:18.800618 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-77b9747d97-tmqrt" event={"ID":"bf2254fd-d416-41c3-a7d5-83ba747117f9","Type":"ContainerStarted","Data":"b3a16a566c1b24b20187045c927b77aeb42a44e2cbaa7da4da8950dd58b40484"} Sep 12 02:26:18.801047 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:18.800704 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-77b9747d97-tmqrt" Sep 12 02:26:18.818830 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:18.818765 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-77b9747d97-tmqrt" podStartSLOduration=2.372890436 podStartE2EDuration="2.818737886s" totalImagesPullingTime="445.84745ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:26:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:26:18.81706771 +0000 UTC m=+2481.609061517" watchObservedRunningTime="2026-09-12 02:26:18.818737886 +0000 UTC m=+2481.610731672" Sep 12 02:26:24.224291 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:24.224260 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:26:24.271281 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:24.271253 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-77b9747d97-tmqrt"] Sep 12 02:26:24.271499 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:24.271470 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-77b9747d97-tmqrt" podUID="bf2254fd-d416-41c3-a7d5-83ba747117f9" containerName="maas-api" containerID="cri-o://b3a16a566c1b24b20187045c927b77aeb42a44e2cbaa7da4da8950dd58b40484" gracePeriod=30 Sep 12 02:26:24.277037 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:24.277008 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-77b9747d97-tmqrt" Sep 12 02:26:24.303720 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:24.303692 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-86c4dff545-28hnx"] Sep 12 02:26:24.307581 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:24.307559 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-86c4dff545-28hnx" Sep 12 02:26:24.323710 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:24.323677 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-86c4dff545-28hnx"] Sep 12 02:26:24.406057 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:24.406025 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54aa083a-6580-45fa-878f-68e8c85d21d5-maas-api-tls\") pod \"maas-api-86c4dff545-28hnx\" (UID: \"54aa083a-6580-45fa-878f-68e8c85d21d5\") " pod="odh-ai-gateway-infra/maas-api-86c4dff545-28hnx" Sep 12 02:26:24.406210 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:24.406066 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55vkj\" (UniqueName: \"kubernetes.io/projected/54aa083a-6580-45fa-878f-68e8c85d21d5-kube-api-access-55vkj\") pod \"maas-api-86c4dff545-28hnx\" (UID: \"54aa083a-6580-45fa-878f-68e8c85d21d5\") " pod="odh-ai-gateway-infra/maas-api-86c4dff545-28hnx" Sep 12 02:26:24.406210 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:24.406112 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54aa083a-6580-45fa-878f-68e8c85d21d5-metrics-certs\") pod \"maas-api-86c4dff545-28hnx\" (UID: \"54aa083a-6580-45fa-878f-68e8c85d21d5\") " pod="odh-ai-gateway-infra/maas-api-86c4dff545-28hnx" Sep 12 02:26:24.406210 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:24.406138 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54aa083a-6580-45fa-878f-68e8c85d21d5-openshift-service-ca\") pod \"maas-api-86c4dff545-28hnx\" (UID: \"54aa083a-6580-45fa-878f-68e8c85d21d5\") " pod="odh-ai-gateway-infra/maas-api-86c4dff545-28hnx" Sep 12 02:26:24.507492 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:24.507429 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54aa083a-6580-45fa-878f-68e8c85d21d5-maas-api-tls\") pod \"maas-api-86c4dff545-28hnx\" (UID: \"54aa083a-6580-45fa-878f-68e8c85d21d5\") " pod="odh-ai-gateway-infra/maas-api-86c4dff545-28hnx" Sep 12 02:26:24.507591 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:24.507512 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55vkj\" (UniqueName: \"kubernetes.io/projected/54aa083a-6580-45fa-878f-68e8c85d21d5-kube-api-access-55vkj\") pod \"maas-api-86c4dff545-28hnx\" (UID: \"54aa083a-6580-45fa-878f-68e8c85d21d5\") " pod="odh-ai-gateway-infra/maas-api-86c4dff545-28hnx" Sep 12 02:26:24.507591 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:24.507573 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54aa083a-6580-45fa-878f-68e8c85d21d5-metrics-certs\") pod \"maas-api-86c4dff545-28hnx\" (UID: \"54aa083a-6580-45fa-878f-68e8c85d21d5\") " pod="odh-ai-gateway-infra/maas-api-86c4dff545-28hnx" Sep 12 02:26:24.507688 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:24.507603 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54aa083a-6580-45fa-878f-68e8c85d21d5-openshift-service-ca\") pod \"maas-api-86c4dff545-28hnx\" (UID: \"54aa083a-6580-45fa-878f-68e8c85d21d5\") " pod="odh-ai-gateway-infra/maas-api-86c4dff545-28hnx" Sep 12 02:26:24.508338 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:24.508307 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54aa083a-6580-45fa-878f-68e8c85d21d5-openshift-service-ca\") pod \"maas-api-86c4dff545-28hnx\" (UID: \"54aa083a-6580-45fa-878f-68e8c85d21d5\") " pod="odh-ai-gateway-infra/maas-api-86c4dff545-28hnx" Sep 12 02:26:24.510041 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:24.510020 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54aa083a-6580-45fa-878f-68e8c85d21d5-maas-api-tls\") pod \"maas-api-86c4dff545-28hnx\" (UID: \"54aa083a-6580-45fa-878f-68e8c85d21d5\") " pod="odh-ai-gateway-infra/maas-api-86c4dff545-28hnx" Sep 12 02:26:24.510129 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:24.510072 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54aa083a-6580-45fa-878f-68e8c85d21d5-metrics-certs\") pod \"maas-api-86c4dff545-28hnx\" (UID: \"54aa083a-6580-45fa-878f-68e8c85d21d5\") " pod="odh-ai-gateway-infra/maas-api-86c4dff545-28hnx" Sep 12 02:26:24.520521 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:24.520495 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55vkj\" (UniqueName: \"kubernetes.io/projected/54aa083a-6580-45fa-878f-68e8c85d21d5-kube-api-access-55vkj\") pod \"maas-api-86c4dff545-28hnx\" (UID: \"54aa083a-6580-45fa-878f-68e8c85d21d5\") " pod="odh-ai-gateway-infra/maas-api-86c4dff545-28hnx" Sep 12 02:26:24.540674 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:24.540650 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-77b9747d97-tmqrt" Sep 12 02:26:24.608151 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:24.608113 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqlv5\" (UniqueName: \"kubernetes.io/projected/bf2254fd-d416-41c3-a7d5-83ba747117f9-kube-api-access-qqlv5\") pod \"bf2254fd-d416-41c3-a7d5-83ba747117f9\" (UID: \"bf2254fd-d416-41c3-a7d5-83ba747117f9\") " Sep 12 02:26:24.608323 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:24.608248 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf2254fd-d416-41c3-a7d5-83ba747117f9-maas-api-tls\") pod \"bf2254fd-d416-41c3-a7d5-83ba747117f9\" (UID: \"bf2254fd-d416-41c3-a7d5-83ba747117f9\") " Sep 12 02:26:24.608323 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:24.608297 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf2254fd-d416-41c3-a7d5-83ba747117f9-metrics-certs\") pod \"bf2254fd-d416-41c3-a7d5-83ba747117f9\" (UID: \"bf2254fd-d416-41c3-a7d5-83ba747117f9\") " Sep 12 02:26:24.608478 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:24.608323 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf2254fd-d416-41c3-a7d5-83ba747117f9-openshift-service-ca\") pod \"bf2254fd-d416-41c3-a7d5-83ba747117f9\" (UID: \"bf2254fd-d416-41c3-a7d5-83ba747117f9\") " Sep 12 02:26:24.608755 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:24.608723 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf2254fd-d416-41c3-a7d5-83ba747117f9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bf2254fd-d416-41c3-a7d5-83ba747117f9" (UID: "bf2254fd-d416-41c3-a7d5-83ba747117f9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 12 02:26:24.610515 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:24.610487 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf2254fd-d416-41c3-a7d5-83ba747117f9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bf2254fd-d416-41c3-a7d5-83ba747117f9" (UID: "bf2254fd-d416-41c3-a7d5-83ba747117f9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 12 02:26:24.610623 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:24.610531 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf2254fd-d416-41c3-a7d5-83ba747117f9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bf2254fd-d416-41c3-a7d5-83ba747117f9" (UID: "bf2254fd-d416-41c3-a7d5-83ba747117f9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 12 02:26:24.610623 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:24.610539 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf2254fd-d416-41c3-a7d5-83ba747117f9-kube-api-access-qqlv5" (OuterVolumeSpecName: "kube-api-access-qqlv5") pod "bf2254fd-d416-41c3-a7d5-83ba747117f9" (UID: "bf2254fd-d416-41c3-a7d5-83ba747117f9"). InnerVolumeSpecName "kube-api-access-qqlv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 12 02:26:24.655583 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:24.655545 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-86c4dff545-28hnx" Sep 12 02:26:24.710322 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:24.710283 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf2254fd-d416-41c3-a7d5-83ba747117f9-maas-api-tls\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:26:24.710439 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:24.710326 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf2254fd-d416-41c3-a7d5-83ba747117f9-metrics-certs\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:26:24.710439 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:24.710341 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf2254fd-d416-41c3-a7d5-83ba747117f9-openshift-service-ca\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:26:24.710439 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:24.710355 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqlv5\" (UniqueName: \"kubernetes.io/projected/bf2254fd-d416-41c3-a7d5-83ba747117f9-kube-api-access-qqlv5\") on node \"ip-10-0-133-79.ec2.internal\" DevicePath \"\"" Sep 12 02:26:24.829586 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:24.829494 2579 generic.go:356] "Generic (PLEG): container finished" podID="bf2254fd-d416-41c3-a7d5-83ba747117f9" containerID="b3a16a566c1b24b20187045c927b77aeb42a44e2cbaa7da4da8950dd58b40484" exitCode=0 Sep 12 02:26:24.829586 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:24.829543 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-77b9747d97-tmqrt" event={"ID":"bf2254fd-d416-41c3-a7d5-83ba747117f9","Type":"ContainerDied","Data":"b3a16a566c1b24b20187045c927b77aeb42a44e2cbaa7da4da8950dd58b40484"} Sep 12 02:26:24.829586 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:24.829586 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-77b9747d97-tmqrt" event={"ID":"bf2254fd-d416-41c3-a7d5-83ba747117f9","Type":"ContainerDied","Data":"3ab17178c1050333cb645323b62580237133cc6036ef0d689381060b69396029"} Sep 12 02:26:24.829863 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:24.829608 2579 scope.go:117] "RemoveContainer" containerID="b3a16a566c1b24b20187045c927b77aeb42a44e2cbaa7da4da8950dd58b40484" Sep 12 02:26:24.829863 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:24.829547 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-77b9747d97-tmqrt" Sep 12 02:26:24.839565 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:24.839546 2579 scope.go:117] "RemoveContainer" containerID="b3a16a566c1b24b20187045c927b77aeb42a44e2cbaa7da4da8950dd58b40484" Sep 12 02:26:24.839883 ip-10-0-133-79 kubenswrapper[2579]: E0912 02:26:24.839860 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3a16a566c1b24b20187045c927b77aeb42a44e2cbaa7da4da8950dd58b40484\": container with ID starting with b3a16a566c1b24b20187045c927b77aeb42a44e2cbaa7da4da8950dd58b40484 not found: ID does not exist" containerID="b3a16a566c1b24b20187045c927b77aeb42a44e2cbaa7da4da8950dd58b40484" Sep 12 02:26:24.839963 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:24.839893 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3a16a566c1b24b20187045c927b77aeb42a44e2cbaa7da4da8950dd58b40484"} err="failed to get container status \"b3a16a566c1b24b20187045c927b77aeb42a44e2cbaa7da4da8950dd58b40484\": rpc error: code = NotFound desc = could not find container \"b3a16a566c1b24b20187045c927b77aeb42a44e2cbaa7da4da8950dd58b40484\": container with ID starting with b3a16a566c1b24b20187045c927b77aeb42a44e2cbaa7da4da8950dd58b40484 not found: ID does not exist" Sep 12 02:26:24.850434 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:24.850402 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-77b9747d97-tmqrt"] Sep 12 02:26:24.852291 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:24.852258 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-77b9747d97-tmqrt"] Sep 12 02:26:24.926392 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:24.926367 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-86c4dff545-28hnx"] Sep 12 02:26:24.928181 ip-10-0-133-79 kubenswrapper[2579]: W0912 02:26:24.928158 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54aa083a_6580_45fa_878f_68e8c85d21d5.slice/crio-9cbde976e6faaad52dc57f4eb7f055bfc69047029e715b132465bf1051375ef2 WatchSource:0}: Error finding container 9cbde976e6faaad52dc57f4eb7f055bfc69047029e715b132465bf1051375ef2: Status 404 returned error can't find the container with id 9cbde976e6faaad52dc57f4eb7f055bfc69047029e715b132465bf1051375ef2 Sep 12 02:26:25.726111 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:25.726077 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf2254fd-d416-41c3-a7d5-83ba747117f9" path="/var/lib/kubelet/pods/bf2254fd-d416-41c3-a7d5-83ba747117f9/volumes" Sep 12 02:26:25.836508 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:25.836468 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-86c4dff545-28hnx" event={"ID":"54aa083a-6580-45fa-878f-68e8c85d21d5","Type":"ContainerStarted","Data":"b6642ef6f2191b7432cc706d7248db33c55ce15ce0cf0a91e3cdb7307d34a19d"} Sep 12 02:26:25.836694 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:25.836531 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-86c4dff545-28hnx" event={"ID":"54aa083a-6580-45fa-878f-68e8c85d21d5","Type":"ContainerStarted","Data":"9cbde976e6faaad52dc57f4eb7f055bfc69047029e715b132465bf1051375ef2"} Sep 12 02:26:25.836694 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:25.836565 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-86c4dff545-28hnx" Sep 12 02:26:25.853466 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:25.853399 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-86c4dff545-28hnx" podStartSLOduration=1.497326421 podStartE2EDuration="1.853383095s" totalImagesPullingTime="356.056674ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:26:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:26:25.852366276 +0000 UTC m=+2488.644360073" watchObservedRunningTime="2026-09-12 02:26:25.853383095 +0000 UTC m=+2488.645376890" Sep 12 02:26:31.223581 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:31.223546 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:26:31.847488 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:31.847443 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-86c4dff545-28hnx" Sep 12 02:26:37.217819 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:37.217786 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:26:44.219349 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:44.219313 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-p5xqc"] Sep 12 02:26:46.723789 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:46.723758 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-56bb445f59-4mxgl_1393128d-7b35-4819-83ee-048251caa3d0/authorino/0.log" Sep 12 02:26:52.921394 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:52.921313 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-578c78994-v7hw8_e61c2cac-5cce-4334-b88d-3677d05fb43a/copy-manifests/0.log" Sep 12 02:26:52.927716 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:52.927694 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-578c78994-v7hw8_e61c2cac-5cce-4334-b88d-3677d05fb43a/manager/0.log" Sep 12 02:26:53.414148 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:53.414120 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-86c4dff545-28hnx_54aa083a-6580-45fa-878f-68e8c85d21d5/maas-api/0.log" Sep 12 02:26:53.657559 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:53.657532 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29819640-868qh_fa80bfbd-b345-47d0-8f68-9c680229276a/cleanup/0.log" Sep 12 02:26:54.679820 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:54.679789 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-56bb445f59-4mxgl_1393128d-7b35-4819-83ee-048251caa3d0/authorino/0.log" Sep 12 02:26:55.286590 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:55.286561 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-p5xqc_a596d221-6246-4003-972e-6f658b315220/limitador/0.log" Sep 12 02:26:56.125625 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:56.125591 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69c99bdc57-jdfsk_53310321-9c0d-4246-9ed7-7ac33f3754b3/kube-auth-proxy/0.log" Sep 12 02:26:57.186145 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:57.186106 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-7bgtp_b5fc4975-3ef2-4488-99ed-1e4f50230ab0/main/0.log" Sep 12 02:26:57.523291 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:57.523205 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-vrrgt_0297867c-0e30-4053-a687-b38a6d25ed52/main/0.log" Sep 12 02:26:57.633750 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:26:57.633715 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-29qsh_3311a5d0-7320-4533-b6dc-56687a3a1028/main/0.log" Sep 12 02:27:03.562846 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:03.562811 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:27:03.663324 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:03.663290 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" limit=1024 Sep 12 02:27:06.401538 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:06.401501 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-wjmgj_f016f46e-b786-405e-8f6e-faca9e639b5f/global-pull-secret-syncer/0.log" Sep 12 02:27:06.447948 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:06.447906 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-nhw89_b1408eab-ddcd-41d6-87af-50980dd43721/konnectivity-agent/0.log" Sep 12 02:27:06.578313 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:06.578275 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-79.ec2.internal_0f092973866d00f19dd94cc0896a8414/haproxy/0.log" Sep 12 02:27:10.471512 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:10.471476 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-56bb445f59-4mxgl_1393128d-7b35-4819-83ee-048251caa3d0/authorino/0.log" Sep 12 02:27:10.690862 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:10.690827 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-p5xqc_a596d221-6246-4003-972e-6f658b315220/limitador/0.log" Sep 12 02:27:12.544051 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:12.544014 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bedf8e26-276b-4209-b34c-77a137548c72/alertmanager/0.log" Sep 12 02:27:12.564810 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:12.564789 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bedf8e26-276b-4209-b34c-77a137548c72/config-reloader/0.log" Sep 12 02:27:12.585251 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:12.585231 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bedf8e26-276b-4209-b34c-77a137548c72/kube-rbac-proxy-web/0.log" Sep 12 02:27:12.604276 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:12.604255 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bedf8e26-276b-4209-b34c-77a137548c72/kube-rbac-proxy/0.log" Sep 12 02:27:12.622005 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:12.621979 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bedf8e26-276b-4209-b34c-77a137548c72/kube-rbac-proxy-metric/0.log" Sep 12 02:27:12.642964 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:12.642947 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bedf8e26-276b-4209-b34c-77a137548c72/prom-label-proxy/0.log" Sep 12 02:27:12.661681 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:12.661658 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bedf8e26-276b-4209-b34c-77a137548c72/init-config-reloader/0.log" Sep 12 02:27:12.725974 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:12.725904 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-cwrwp_41d0d41a-0c50-40f7-b01c-99bdfb61cc18/kube-state-metrics/0.log" Sep 12 02:27:12.742603 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:12.742578 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-cwrwp_41d0d41a-0c50-40f7-b01c-99bdfb61cc18/kube-rbac-proxy-main/0.log" Sep 12 02:27:12.759846 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:12.759830 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-cwrwp_41d0d41a-0c50-40f7-b01c-99bdfb61cc18/kube-rbac-proxy-self/0.log" Sep 12 02:27:12.784592 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:12.784576 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6ff6c7b584-hbbsv_01af4e11-0c36-4ddb-afa6-79acbc8ec2c1/metrics-server/0.log" Sep 12 02:27:12.812431 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:12.812397 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-bnt6v_81ea416f-0dd9-46dc-8c78-ab1a4360ec5f/monitoring-plugin/0.log" Sep 12 02:27:12.840147 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:12.840119 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8nbjq_f8921497-7f53-4874-ba19-3528fe3f7cd3/node-exporter/0.log" Sep 12 02:27:12.864151 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:12.864112 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8nbjq_f8921497-7f53-4874-ba19-3528fe3f7cd3/kube-rbac-proxy/0.log" Sep 12 02:27:12.893725 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:12.893707 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8nbjq_f8921497-7f53-4874-ba19-3528fe3f7cd3/init-textfile/0.log" Sep 12 02:27:13.139756 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:13.139729 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7f40b713-de7e-443d-bd28-0bf631ee73b3/prometheus/0.log" Sep 12 02:27:13.162954 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:13.162928 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7f40b713-de7e-443d-bd28-0bf631ee73b3/config-reloader/0.log" Sep 12 02:27:13.182698 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:13.182672 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7f40b713-de7e-443d-bd28-0bf631ee73b3/thanos-sidecar/0.log" Sep 12 02:27:13.203740 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:13.203718 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7f40b713-de7e-443d-bd28-0bf631ee73b3/kube-rbac-proxy-web/0.log" Sep 12 02:27:13.224414 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:13.224393 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7f40b713-de7e-443d-bd28-0bf631ee73b3/kube-rbac-proxy/0.log" Sep 12 02:27:13.242996 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:13.242979 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7f40b713-de7e-443d-bd28-0bf631ee73b3/kube-rbac-proxy-thanos/0.log" Sep 12 02:27:13.261523 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:13.261501 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7f40b713-de7e-443d-bd28-0bf631ee73b3/init-config-reloader/0.log" Sep 12 02:27:13.286769 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:13.286744 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-m6t8r_4d37e287-c2ca-4776-bac4-b52132a2ad57/prometheus-operator/0.log" Sep 12 02:27:13.310044 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:13.310011 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-m6t8r_4d37e287-c2ca-4776-bac4-b52132a2ad57/kube-rbac-proxy/0.log" Sep 12 02:27:13.332447 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:13.332413 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-q9jhh_c3d9610e-3251-4262-918f-12f28f4aef86/prometheus-operator-admission-webhook/0.log" Sep 12 02:27:13.381525 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:13.381502 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-675d55d6d-6rkn4_f746c621-3575-496b-88b5-eaf926350ab1/telemeter-client/0.log" Sep 12 02:27:13.401913 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:13.401838 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-675d55d6d-6rkn4_f746c621-3575-496b-88b5-eaf926350ab1/reload/0.log" Sep 12 02:27:13.425162 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:13.425138 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-675d55d6d-6rkn4_f746c621-3575-496b-88b5-eaf926350ab1/kube-rbac-proxy/0.log" Sep 12 02:27:13.467346 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:13.467319 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-f67ddbc77-l6kn6_7905e746-9f1b-41cb-af97-43f87dc75594/thanos-query/0.log" Sep 12 02:27:13.489240 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:13.489216 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-f67ddbc77-l6kn6_7905e746-9f1b-41cb-af97-43f87dc75594/kube-rbac-proxy-web/0.log" Sep 12 02:27:13.513511 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:13.513481 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-f67ddbc77-l6kn6_7905e746-9f1b-41cb-af97-43f87dc75594/kube-rbac-proxy/0.log" Sep 12 02:27:13.538393 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:13.538368 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-f67ddbc77-l6kn6_7905e746-9f1b-41cb-af97-43f87dc75594/prom-label-proxy/0.log" Sep 12 02:27:13.568304 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:13.568283 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-f67ddbc77-l6kn6_7905e746-9f1b-41cb-af97-43f87dc75594/kube-rbac-proxy-rules/0.log" Sep 12 02:27:13.592298 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:13.592277 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-f67ddbc77-l6kn6_7905e746-9f1b-41cb-af97-43f87dc75594/kube-rbac-proxy-metrics/0.log" Sep 12 02:27:15.164587 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:15.164558 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-nnp46_af0906c4-27c2-4ff2-8c04-7f121cc59a3a/networking-console-plugin/0.log" Sep 12 02:27:16.566304 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:16.566270 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-mgsgx_9fbbd8af-609e-4eac-927a-0d90c136858b/download-server/0.log" Sep 12 02:27:17.073735 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:17.073693 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-59ltq/perf-node-gather-daemonset-cz87d"] Sep 12 02:27:17.074273 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:17.074253 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf2254fd-d416-41c3-a7d5-83ba747117f9" containerName="maas-api" Sep 12 02:27:17.074273 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:17.074273 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf2254fd-d416-41c3-a7d5-83ba747117f9" containerName="maas-api" Sep 12 02:27:17.074436 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:17.074355 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf2254fd-d416-41c3-a7d5-83ba747117f9" containerName="maas-api" Sep 12 02:27:17.077742 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:17.077725 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-59ltq/perf-node-gather-daemonset-cz87d" Sep 12 02:27:17.080028 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:17.080004 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-59ltq\"/\"kube-root-ca.crt\"" Sep 12 02:27:17.080831 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:17.080809 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-59ltq\"/\"openshift-service-ca.crt\"" Sep 12 02:27:17.080974 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:17.080866 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-59ltq\"/\"default-dockercfg-pg6t6\"" Sep 12 02:27:17.089267 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:17.088963 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-59ltq/perf-node-gather-daemonset-cz87d"] Sep 12 02:27:17.210805 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:17.210779 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96lvz\" (UniqueName: \"kubernetes.io/projected/31bfbbc6-c20f-4916-bd12-f64d981f9ed7-kube-api-access-96lvz\") pod \"perf-node-gather-daemonset-cz87d\" (UID: \"31bfbbc6-c20f-4916-bd12-f64d981f9ed7\") " pod="openshift-must-gather-59ltq/perf-node-gather-daemonset-cz87d" Sep 12 02:27:17.210986 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:17.210841 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/31bfbbc6-c20f-4916-bd12-f64d981f9ed7-proc\") pod \"perf-node-gather-daemonset-cz87d\" (UID: \"31bfbbc6-c20f-4916-bd12-f64d981f9ed7\") " pod="openshift-must-gather-59ltq/perf-node-gather-daemonset-cz87d" Sep 12 02:27:17.210986 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:17.210861 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/31bfbbc6-c20f-4916-bd12-f64d981f9ed7-sys\") pod \"perf-node-gather-daemonset-cz87d\" (UID: \"31bfbbc6-c20f-4916-bd12-f64d981f9ed7\") " pod="openshift-must-gather-59ltq/perf-node-gather-daemonset-cz87d" Sep 12 02:27:17.210986 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:17.210907 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/31bfbbc6-c20f-4916-bd12-f64d981f9ed7-lib-modules\") pod \"perf-node-gather-daemonset-cz87d\" (UID: \"31bfbbc6-c20f-4916-bd12-f64d981f9ed7\") " pod="openshift-must-gather-59ltq/perf-node-gather-daemonset-cz87d" Sep 12 02:27:17.210986 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:17.210926 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/31bfbbc6-c20f-4916-bd12-f64d981f9ed7-podres\") pod \"perf-node-gather-daemonset-cz87d\" (UID: \"31bfbbc6-c20f-4916-bd12-f64d981f9ed7\") " pod="openshift-must-gather-59ltq/perf-node-gather-daemonset-cz87d" Sep 12 02:27:17.311780 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:17.311744 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96lvz\" (UniqueName: \"kubernetes.io/projected/31bfbbc6-c20f-4916-bd12-f64d981f9ed7-kube-api-access-96lvz\") pod \"perf-node-gather-daemonset-cz87d\" (UID: \"31bfbbc6-c20f-4916-bd12-f64d981f9ed7\") " pod="openshift-must-gather-59ltq/perf-node-gather-daemonset-cz87d" Sep 12 02:27:17.311942 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:17.311812 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/31bfbbc6-c20f-4916-bd12-f64d981f9ed7-proc\") pod \"perf-node-gather-daemonset-cz87d\" (UID: \"31bfbbc6-c20f-4916-bd12-f64d981f9ed7\") " pod="openshift-must-gather-59ltq/perf-node-gather-daemonset-cz87d" Sep 12 02:27:17.311942 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:17.311831 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/31bfbbc6-c20f-4916-bd12-f64d981f9ed7-sys\") pod \"perf-node-gather-daemonset-cz87d\" (UID: \"31bfbbc6-c20f-4916-bd12-f64d981f9ed7\") " pod="openshift-must-gather-59ltq/perf-node-gather-daemonset-cz87d" Sep 12 02:27:17.311942 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:17.311856 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/31bfbbc6-c20f-4916-bd12-f64d981f9ed7-lib-modules\") pod \"perf-node-gather-daemonset-cz87d\" (UID: \"31bfbbc6-c20f-4916-bd12-f64d981f9ed7\") " pod="openshift-must-gather-59ltq/perf-node-gather-daemonset-cz87d" Sep 12 02:27:17.311942 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:17.311874 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/31bfbbc6-c20f-4916-bd12-f64d981f9ed7-podres\") pod \"perf-node-gather-daemonset-cz87d\" (UID: \"31bfbbc6-c20f-4916-bd12-f64d981f9ed7\") " pod="openshift-must-gather-59ltq/perf-node-gather-daemonset-cz87d" Sep 12 02:27:17.312122 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:17.311936 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/31bfbbc6-c20f-4916-bd12-f64d981f9ed7-sys\") pod \"perf-node-gather-daemonset-cz87d\" (UID: \"31bfbbc6-c20f-4916-bd12-f64d981f9ed7\") " pod="openshift-must-gather-59ltq/perf-node-gather-daemonset-cz87d" Sep 12 02:27:17.312122 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:17.311992 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/31bfbbc6-c20f-4916-bd12-f64d981f9ed7-podres\") pod \"perf-node-gather-daemonset-cz87d\" (UID: \"31bfbbc6-c20f-4916-bd12-f64d981f9ed7\") " pod="openshift-must-gather-59ltq/perf-node-gather-daemonset-cz87d" Sep 12 02:27:17.312122 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:17.311990 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/31bfbbc6-c20f-4916-bd12-f64d981f9ed7-proc\") pod \"perf-node-gather-daemonset-cz87d\" (UID: \"31bfbbc6-c20f-4916-bd12-f64d981f9ed7\") " pod="openshift-must-gather-59ltq/perf-node-gather-daemonset-cz87d" Sep 12 02:27:17.312122 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:17.312020 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/31bfbbc6-c20f-4916-bd12-f64d981f9ed7-lib-modules\") pod \"perf-node-gather-daemonset-cz87d\" (UID: \"31bfbbc6-c20f-4916-bd12-f64d981f9ed7\") " pod="openshift-must-gather-59ltq/perf-node-gather-daemonset-cz87d" Sep 12 02:27:17.319404 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:17.319382 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96lvz\" (UniqueName: \"kubernetes.io/projected/31bfbbc6-c20f-4916-bd12-f64d981f9ed7-kube-api-access-96lvz\") pod \"perf-node-gather-daemonset-cz87d\" (UID: \"31bfbbc6-c20f-4916-bd12-f64d981f9ed7\") " pod="openshift-must-gather-59ltq/perf-node-gather-daemonset-cz87d" Sep 12 02:27:17.394216 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:17.394188 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-59ltq/perf-node-gather-daemonset-cz87d" Sep 12 02:27:17.669121 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:17.669043 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-59ltq/perf-node-gather-daemonset-cz87d"] Sep 12 02:27:17.671886 ip-10-0-133-79 kubenswrapper[2579]: W0912 02:27:17.671857 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod31bfbbc6_c20f_4916_bd12_f64d981f9ed7.slice/crio-98cc6354ffada29f1abe3f5aa3738acab831dc8e450502fb99d4451439126aa9 WatchSource:0}: Error finding container 98cc6354ffada29f1abe3f5aa3738acab831dc8e450502fb99d4451439126aa9: Status 404 returned error can't find the container with id 98cc6354ffada29f1abe3f5aa3738acab831dc8e450502fb99d4451439126aa9 Sep 12 02:27:18.104141 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:18.104048 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-59ltq/perf-node-gather-daemonset-cz87d" event={"ID":"31bfbbc6-c20f-4916-bd12-f64d981f9ed7","Type":"ContainerStarted","Data":"2919535b657fe19235d75b5d7b18b75e9e5d40d93cf373b202dc0b621beedea1"} Sep 12 02:27:18.104141 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:18.104096 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-59ltq/perf-node-gather-daemonset-cz87d" event={"ID":"31bfbbc6-c20f-4916-bd12-f64d981f9ed7","Type":"ContainerStarted","Data":"98cc6354ffada29f1abe3f5aa3738acab831dc8e450502fb99d4451439126aa9"} Sep 12 02:27:18.104337 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:18.104153 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-59ltq/perf-node-gather-daemonset-cz87d" Sep 12 02:27:18.119968 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:18.119922 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-59ltq/perf-node-gather-daemonset-cz87d" podStartSLOduration=1.1199086679999999 podStartE2EDuration="1.119908668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 02:27:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:27:18.11818295 +0000 UTC m=+2540.910176746" watchObservedRunningTime="2026-09-12 02:27:18.119908668 +0000 UTC m=+2540.911902463" Sep 12 02:27:18.573339 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:18.573312 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-p96nw_f2d765cc-6edd-4ba0-b8b0-81391dd5aa3d/dns/0.log" Sep 12 02:27:18.592205 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:18.592181 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-p96nw_f2d765cc-6edd-4ba0-b8b0-81391dd5aa3d/kube-rbac-proxy/0.log" Sep 12 02:27:18.634188 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:18.634166 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qzpw5_a1140d6e-61e4-4bad-a337-cc28db7dd812/dns-node-resolver/0.log" Sep 12 02:27:19.195198 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:19.195170 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7dddc48645-jt7vp_6b452077-08b8-4f90-a95a-85e0d51d9e4a/registry/0.log" Sep 12 02:27:19.214604 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:19.214588 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2dpmw_6e1eb253-a93e-4370-9cbd-a5ec2ae013a9/node-ca/0.log" Sep 12 02:27:20.989906 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:20.989881 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69c99bdc57-jdfsk_53310321-9c0d-4246-9ed7-7ac33f3754b3/kube-auth-proxy/0.log" Sep 12 02:27:21.988546 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:21.988501 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8bm2n_c84a24ed-159c-4497-863f-03a5a9c0d81d/serve-healthcheck-canary/0.log" Sep 12 02:27:22.498282 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:22.498255 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gdkkk_7d6c62d0-83e1-41e7-8780-5dc9be2074a5/kube-rbac-proxy/0.log" Sep 12 02:27:22.516338 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:22.516318 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gdkkk_7d6c62d0-83e1-41e7-8780-5dc9be2074a5/exporter/0.log" Sep 12 02:27:22.535306 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:22.535285 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gdkkk_7d6c62d0-83e1-41e7-8780-5dc9be2074a5/extractor/0.log" Sep 12 02:27:24.123243 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:24.123214 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-59ltq/perf-node-gather-daemonset-cz87d" Sep 12 02:27:25.354910 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:25.354885 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-578c78994-v7hw8_e61c2cac-5cce-4334-b88d-3677d05fb43a/manager/0.log" Sep 12 02:27:25.373887 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:25.373865 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-578c78994-v7hw8_e61c2cac-5cce-4334-b88d-3677d05fb43a/copy-manifests/0.log" Sep 12 02:27:26.925071 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:26.925041 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-b9f9494c8-kjlfm_ae104ce1-96f0-4774-bb2d-e17e478b4afd/manager/0.log" Sep 12 02:27:28.080799 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:28.080775 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69c99bdc57-jdfsk_53310321-9c0d-4246-9ed7-7ac33f3754b3/kube-auth-proxy/0.log" Sep 12 02:27:30.610659 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:30.610634 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69c99bdc57-jdfsk_53310321-9c0d-4246-9ed7-7ac33f3754b3/kube-auth-proxy/0.log" Sep 12 02:27:31.889194 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:31.889168 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-bshhm_e083dae5-f950-4f0d-bbc0-4511763aeee2/migrator/0.log" Sep 12 02:27:31.906942 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:31.906918 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-bshhm_e083dae5-f950-4f0d-bbc0-4511763aeee2/graceful-termination/0.log" Sep 12 02:27:33.424134 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:33.424098 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rzbs6_46256f89-0730-441e-97d5-6ef1dff74569/kube-multus-additional-cni-plugins/0.log" Sep 12 02:27:33.444252 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:33.444230 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rzbs6_46256f89-0730-441e-97d5-6ef1dff74569/egress-router-binary-copy/0.log" Sep 12 02:27:33.463552 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:33.463531 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rzbs6_46256f89-0730-441e-97d5-6ef1dff74569/cni-plugins/0.log" Sep 12 02:27:33.481651 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:33.481620 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rzbs6_46256f89-0730-441e-97d5-6ef1dff74569/bond-cni-plugin/0.log" Sep 12 02:27:33.500580 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:33.500561 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rzbs6_46256f89-0730-441e-97d5-6ef1dff74569/routeoverride-cni/0.log" Sep 12 02:27:33.518636 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:33.518619 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rzbs6_46256f89-0730-441e-97d5-6ef1dff74569/whereabouts-cni-bincopy/0.log" Sep 12 02:27:33.536824 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:33.536806 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rzbs6_46256f89-0730-441e-97d5-6ef1dff74569/whereabouts-cni/0.log" Sep 12 02:27:33.799613 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:33.799538 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-d4598_e5ed1052-5fa4-4d50-be77-d799d5722140/kube-multus/0.log" Sep 12 02:27:33.871967 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:33.871941 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-nhq7s_b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03/network-metrics-daemon/0.log" Sep 12 02:27:33.888717 ip-10-0-133-79 kubenswrapper[2579]: I0912 02:27:33.888679 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-nhq7s_b79c3e7e-cd1c-4833-a3cf-78e6bd3b8f03/kube-rbac-proxy/0.log"