Sep 11 19:58:32.205227 ip-10-0-138-205 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 11 19:58:32.205238 ip-10-0-138-205 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 11 19:58:32.205245 ip-10-0-138-205 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 11 19:58:32.205393 ip-10-0-138-205 systemd[1]: Failed to start Kubernetes Kubelet. Sep 11 19:58:42.371262 ip-10-0-138-205 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 11 19:58:42.371278 ip-10-0-138-205 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot ea4b23c5846a42b3b1b6a3e274472ce6 -- Sep 11 20:00:28.664310 ip-10-0-138-205 systemd[1]: Starting Kubernetes Kubelet... Sep 11 20:00:29.084211 ip-10-0-138-205 kubenswrapper[2546]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 20:00:29.084766 ip-10-0-138-205 kubenswrapper[2546]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 11 20:00:29.084766 ip-10-0-138-205 kubenswrapper[2546]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 20:00:29.084766 ip-10-0-138-205 kubenswrapper[2546]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 11 20:00:29.084766 ip-10-0-138-205 kubenswrapper[2546]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 20:00:29.085356 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.085275 2546 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 11 20:00:29.089409 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089394 2546 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 20:00:29.089409 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089408 2546 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 20:00:29.089472 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089412 2546 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 20:00:29.089472 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089416 2546 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 20:00:29.089472 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089419 2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 20:00:29.089472 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089422 2546 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 20:00:29.089472 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089426 2546 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 20:00:29.089472 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089428 2546 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 20:00:29.089472 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089431 2546 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 20:00:29.089472 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089434 2546 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 20:00:29.089472 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089436 2546 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 20:00:29.089472 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089439 2546 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 20:00:29.089472 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089442 2546 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 20:00:29.089472 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089444 2546 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 20:00:29.089472 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089447 2546 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 20:00:29.089472 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089449 2546 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 20:00:29.089472 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089452 2546 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 20:00:29.089472 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089454 2546 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 20:00:29.089472 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089457 2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 20:00:29.089472 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089460 2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 20:00:29.089472 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089463 2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 20:00:29.089472 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089465 2546 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 20:00:29.089958 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089468 2546 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 20:00:29.089958 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089470 2546 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 20:00:29.089958 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089473 2546 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 20:00:29.089958 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089476 2546 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 20:00:29.089958 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089479 2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 20:00:29.089958 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089483 2546 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 20:00:29.089958 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089487 2546 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 20:00:29.089958 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089490 2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 20:00:29.089958 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089493 2546 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 20:00:29.089958 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089496 2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 20:00:29.089958 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089499 2546 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 20:00:29.089958 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089501 2546 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 20:00:29.089958 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089504 2546 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 20:00:29.089958 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089507 2546 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 20:00:29.089958 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089530 2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 20:00:29.089958 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089534 2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 20:00:29.089958 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089537 2546 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 20:00:29.089958 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089540 2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 20:00:29.089958 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089543 2546 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 20:00:29.090460 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089546 2546 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 20:00:29.090460 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089549 2546 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 20:00:29.090460 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089552 2546 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 20:00:29.090460 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089565 2546 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 20:00:29.090460 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089569 2546 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 20:00:29.090460 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089571 2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 20:00:29.090460 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089574 2546 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 20:00:29.090460 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089576 2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 20:00:29.090460 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089606 2546 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 20:00:29.090460 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089611 2546 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 20:00:29.090460 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089614 2546 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 20:00:29.090460 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089616 2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 20:00:29.090460 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089619 2546 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 20:00:29.090460 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089621 2546 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 20:00:29.090460 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089625 2546 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 20:00:29.090460 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089629 2546 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 20:00:29.090460 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089633 2546 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 20:00:29.090460 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089636 2546 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 20:00:29.090460 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089640 2546 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 20:00:29.090931 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089643 2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 20:00:29.090931 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089645 2546 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 20:00:29.090931 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089648 2546 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 20:00:29.090931 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089650 2546 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 20:00:29.090931 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089653 2546 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 20:00:29.090931 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089656 2546 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 20:00:29.090931 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089658 2546 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 20:00:29.090931 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089661 2546 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 20:00:29.090931 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089664 2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 20:00:29.090931 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089666 2546 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 20:00:29.090931 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089669 2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 20:00:29.090931 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089671 2546 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 20:00:29.090931 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089674 2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 20:00:29.090931 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089676 2546 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 20:00:29.090931 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089679 2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 20:00:29.090931 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089682 2546 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 20:00:29.090931 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089684 2546 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 20:00:29.090931 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089686 2546 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 20:00:29.090931 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089689 2546 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 20:00:29.090931 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089692 2546 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 20:00:29.091466 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089695 2546 feature_gate.go:328] unrecognized feature gate: Example Sep 11 20:00:29.091466 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089698 2546 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 20:00:29.091466 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089706 2546 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 20:00:29.091466 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089710 2546 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 20:00:29.091466 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089712 2546 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 20:00:29.091466 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089715 2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 20:00:29.091466 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089719 2546 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 20:00:29.091466 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089722 2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 20:00:29.091466 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089725 2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 20:00:29.091466 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089728 2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 20:00:29.091466 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089731 2546 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 20:00:29.091466 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089733 2546 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 20:00:29.091466 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089736 2546 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 20:00:29.091466 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089738 2546 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 20:00:29.091466 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089741 2546 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 20:00:29.091466 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089743 2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 20:00:29.091466 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089746 2546 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 20:00:29.091466 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089749 2546 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 20:00:29.091466 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.089751 2546 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 20:00:29.091466 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090474 2546 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 20:00:29.091958 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090480 2546 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 20:00:29.091958 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090483 2546 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 20:00:29.091958 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090486 2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 20:00:29.091958 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090489 2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 20:00:29.091958 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090491 2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 20:00:29.091958 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090494 2546 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 20:00:29.091958 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090498 2546 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 20:00:29.091958 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090503 2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 20:00:29.091958 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090506 2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 20:00:29.091958 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090508 2546 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 20:00:29.091958 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090511 2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 20:00:29.091958 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090514 2546 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 20:00:29.091958 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090517 2546 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 20:00:29.091958 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090519 2546 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 20:00:29.091958 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090522 2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 20:00:29.091958 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090526 2546 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 20:00:29.091958 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090529 2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 20:00:29.091958 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090532 2546 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 20:00:29.091958 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090534 2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 20:00:29.092444 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090537 2546 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 20:00:29.092444 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090540 2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 20:00:29.092444 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090543 2546 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 20:00:29.092444 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090545 2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 20:00:29.092444 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090548 2546 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 20:00:29.092444 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090551 2546 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 20:00:29.092444 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090553 2546 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 20:00:29.092444 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090556 2546 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 20:00:29.092444 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090558 2546 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 20:00:29.092444 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090561 2546 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 20:00:29.092444 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090564 2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 20:00:29.092444 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090566 2546 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 20:00:29.092444 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090569 2546 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 20:00:29.092444 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090572 2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 20:00:29.092444 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090574 2546 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 20:00:29.092444 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090577 2546 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 20:00:29.092444 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090579 2546 feature_gate.go:328] unrecognized feature gate: Example Sep 11 20:00:29.092444 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090582 2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 20:00:29.092444 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090585 2546 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 20:00:29.092444 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090587 2546 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 20:00:29.092936 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090590 2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 20:00:29.092936 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090592 2546 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 20:00:29.092936 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090595 2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 20:00:29.092936 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090597 2546 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 20:00:29.092936 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090600 2546 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 20:00:29.092936 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090602 2546 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 20:00:29.092936 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090605 2546 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 20:00:29.092936 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090607 2546 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 20:00:29.092936 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090613 2546 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 20:00:29.092936 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090617 2546 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 20:00:29.092936 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090620 2546 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 20:00:29.092936 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090622 2546 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 20:00:29.092936 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090627 2546 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 20:00:29.092936 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090631 2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 20:00:29.092936 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090634 2546 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 20:00:29.092936 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090637 2546 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 20:00:29.092936 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090641 2546 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 20:00:29.092936 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090644 2546 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 20:00:29.092936 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090646 2546 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 20:00:29.093417 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090649 2546 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 20:00:29.093417 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090652 2546 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 20:00:29.093417 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090655 2546 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 20:00:29.093417 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090657 2546 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 20:00:29.093417 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090660 2546 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 20:00:29.093417 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090662 2546 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 20:00:29.093417 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090665 2546 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 20:00:29.093417 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090667 2546 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 20:00:29.093417 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090670 2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 20:00:29.093417 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090673 2546 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 20:00:29.093417 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090675 2546 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 20:00:29.093417 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090678 2546 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 20:00:29.093417 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090680 2546 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 20:00:29.093417 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090683 2546 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 20:00:29.093417 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090685 2546 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 20:00:29.093417 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090688 2546 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 20:00:29.093417 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090691 2546 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 20:00:29.093417 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090694 2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 20:00:29.093417 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090696 2546 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 20:00:29.093417 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090699 2546 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 20:00:29.093905 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090702 2546 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 20:00:29.093905 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090705 2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 20:00:29.093905 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090708 2546 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 20:00:29.093905 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090710 2546 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 20:00:29.093905 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090713 2546 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 20:00:29.093905 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090716 2546 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 20:00:29.093905 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090718 2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 20:00:29.093905 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090721 2546 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 20:00:29.093905 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090723 2546 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 20:00:29.093905 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090726 2546 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 20:00:29.093905 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090728 2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 20:00:29.093905 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090731 2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 20:00:29.093905 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090734 2546 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 20:00:29.093905 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090736 2546 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 20:00:29.093905 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090739 2546 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 20:00:29.093905 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090741 2546 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 20:00:29.093905 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090744 2546 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 20:00:29.093905 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090746 2546 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 20:00:29.093905 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090749 2546 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 20:00:29.093905 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.090751 2546 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 20:00:29.094411 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091117 2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 20:00:29.094411 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091123 2546 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 20:00:29.094411 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091126 2546 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 20:00:29.094411 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091129 2546 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 20:00:29.094411 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091134 2546 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 20:00:29.094411 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091150 2546 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 20:00:29.094411 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091153 2546 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 20:00:29.094411 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091157 2546 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 20:00:29.094411 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091160 2546 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 20:00:29.094411 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091163 2546 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 20:00:29.094411 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091166 2546 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 20:00:29.094411 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091169 2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 20:00:29.094411 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091172 2546 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 20:00:29.094411 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091175 2546 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 20:00:29.094411 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091178 2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 20:00:29.094411 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091181 2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 20:00:29.094411 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091184 2546 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 20:00:29.094411 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091188 2546 feature_gate.go:328] unrecognized feature gate: Example Sep 11 20:00:29.094411 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091191 2546 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 20:00:29.094411 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091194 2546 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 20:00:29.094916 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091197 2546 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 20:00:29.094916 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091200 2546 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 20:00:29.094916 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091203 2546 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 20:00:29.094916 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091206 2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 20:00:29.094916 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091209 2546 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 20:00:29.094916 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091211 2546 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 20:00:29.094916 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091214 2546 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 20:00:29.094916 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091216 2546 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 20:00:29.094916 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091219 2546 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 20:00:29.094916 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091221 2546 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 20:00:29.094916 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091224 2546 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 20:00:29.094916 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091226 2546 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 20:00:29.094916 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091229 2546 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 20:00:29.094916 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091231 2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 20:00:29.094916 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091234 2546 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 20:00:29.094916 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091236 2546 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 20:00:29.094916 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091240 2546 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 20:00:29.094916 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091243 2546 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 20:00:29.094916 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091245 2546 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 20:00:29.094916 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091248 2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 20:00:29.095417 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091251 2546 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 20:00:29.095417 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091253 2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 20:00:29.095417 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091256 2546 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 20:00:29.095417 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091259 2546 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 20:00:29.095417 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091261 2546 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 20:00:29.095417 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091264 2546 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 20:00:29.095417 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091266 2546 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 20:00:29.095417 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091269 2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 20:00:29.095417 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091272 2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 20:00:29.095417 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091275 2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 20:00:29.095417 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091278 2546 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 20:00:29.095417 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091280 2546 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 20:00:29.095417 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091283 2546 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 20:00:29.095417 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091285 2546 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 20:00:29.095417 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091288 2546 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 20:00:29.095417 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091290 2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 20:00:29.095417 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091293 2546 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 20:00:29.095417 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091295 2546 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 20:00:29.095417 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091298 2546 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 20:00:29.095417 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091300 2546 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 20:00:29.095920 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091304 2546 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 20:00:29.095920 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091307 2546 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 20:00:29.095920 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091310 2546 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 20:00:29.095920 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091313 2546 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 20:00:29.095920 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091316 2546 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 20:00:29.095920 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091318 2546 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 20:00:29.095920 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091321 2546 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 20:00:29.095920 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091324 2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 20:00:29.095920 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091327 2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 20:00:29.095920 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091330 2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 20:00:29.095920 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091332 2546 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 20:00:29.095920 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091335 2546 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 20:00:29.095920 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091337 2546 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 20:00:29.095920 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091340 2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 20:00:29.095920 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091342 2546 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 20:00:29.095920 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091346 2546 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 20:00:29.095920 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091350 2546 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 20:00:29.095920 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091353 2546 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 20:00:29.095920 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091357 2546 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 20:00:29.096429 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091361 2546 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 20:00:29.096429 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091364 2546 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 20:00:29.096429 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091367 2546 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 20:00:29.096429 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091370 2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 20:00:29.096429 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091373 2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 20:00:29.096429 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091375 2546 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 20:00:29.096429 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091378 2546 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 20:00:29.096429 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091380 2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 20:00:29.096429 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091383 2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 20:00:29.096429 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091385 2546 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 20:00:29.096429 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091387 2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 20:00:29.096429 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091390 2546 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 20:00:29.096429 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091392 2546 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 20:00:29.096429 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091395 2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 20:00:29.096429 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091397 2546 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 20:00:29.096429 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091400 2546 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 20:00:29.096429 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091402 2546 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 20:00:29.096429 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091405 2546 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 20:00:29.096429 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091407 2546 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 20:00:29.096429 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.091410 2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 20:00:29.096922 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091478 2546 flags.go:64] FLAG: --address="0.0.0.0" Sep 11 20:00:29.096922 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091525 2546 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 11 20:00:29.096922 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091534 2546 flags.go:64] FLAG: --anonymous-auth="true" Sep 11 20:00:29.096922 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091538 2546 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 11 20:00:29.096922 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091542 2546 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 11 20:00:29.096922 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091545 2546 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 11 20:00:29.096922 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091549 2546 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 11 20:00:29.096922 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091553 2546 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 11 20:00:29.096922 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091557 2546 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 11 20:00:29.096922 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091559 2546 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 11 20:00:29.096922 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091563 2546 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 11 20:00:29.096922 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091566 2546 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 11 20:00:29.096922 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091569 2546 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 11 20:00:29.096922 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091572 2546 flags.go:64] FLAG: --cgroup-root="" Sep 11 20:00:29.096922 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091577 2546 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 11 20:00:29.096922 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091580 2546 flags.go:64] FLAG: --client-ca-file="" Sep 11 20:00:29.096922 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091583 2546 flags.go:64] FLAG: --cloud-provider="external" Sep 11 20:00:29.096922 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091586 2546 flags.go:64] FLAG: --cluster-dns="[]" Sep 11 20:00:29.096922 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091591 2546 flags.go:64] FLAG: --cluster-domain="" Sep 11 20:00:29.096922 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091594 2546 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 11 20:00:29.096922 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091597 2546 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 11 20:00:29.096922 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091600 2546 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 11 20:00:29.096922 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091603 2546 flags.go:64] FLAG: --container-log-max-files="5" Sep 11 20:00:29.096922 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091607 2546 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 11 20:00:29.097594 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091610 2546 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 11 20:00:29.097594 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091613 2546 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 11 20:00:29.097594 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091616 2546 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 11 20:00:29.097594 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091619 2546 flags.go:64] FLAG: --contention-profiling="false" Sep 11 20:00:29.097594 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091622 2546 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 11 20:00:29.097594 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091625 2546 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 11 20:00:29.097594 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091629 2546 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 11 20:00:29.097594 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091632 2546 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 11 20:00:29.097594 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091636 2546 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 11 20:00:29.097594 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091639 2546 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 11 20:00:29.097594 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091642 2546 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 11 20:00:29.097594 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091645 2546 flags.go:64] FLAG: --enable-load-reader="false" Sep 11 20:00:29.097594 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091648 2546 flags.go:64] FLAG: --enable-server="true" Sep 11 20:00:29.097594 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091651 2546 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 11 20:00:29.097594 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091656 2546 flags.go:64] FLAG: --event-burst="100" Sep 11 20:00:29.097594 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091659 2546 flags.go:64] FLAG: --event-qps="50" Sep 11 20:00:29.097594 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091662 2546 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 11 20:00:29.097594 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091665 2546 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 11 20:00:29.097594 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091668 2546 flags.go:64] FLAG: --eviction-hard="" Sep 11 20:00:29.097594 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091672 2546 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 11 20:00:29.097594 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091682 2546 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 11 20:00:29.097594 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091686 2546 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 11 20:00:29.097594 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091691 2546 flags.go:64] FLAG: --eviction-soft="" Sep 11 20:00:29.097594 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091694 2546 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 11 20:00:29.097594 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091697 2546 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 11 20:00:29.098253 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091700 2546 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 11 20:00:29.098253 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091703 2546 flags.go:64] FLAG: --experimental-mounter-path="" Sep 11 20:00:29.098253 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091706 2546 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 11 20:00:29.098253 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091710 2546 flags.go:64] FLAG: --fail-swap-on="true" Sep 11 20:00:29.098253 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091713 2546 flags.go:64] FLAG: --feature-gates="" Sep 11 20:00:29.098253 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091722 2546 flags.go:64] FLAG: --file-check-frequency="20s" Sep 11 20:00:29.098253 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091725 2546 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 11 20:00:29.098253 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091728 2546 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 11 20:00:29.098253 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091731 2546 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 11 20:00:29.098253 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091734 2546 flags.go:64] FLAG: --healthz-port="10248" Sep 11 20:00:29.098253 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091737 2546 flags.go:64] FLAG: --help="false" Sep 11 20:00:29.098253 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091740 2546 flags.go:64] FLAG: --hostname-override="ip-10-0-138-205.ec2.internal" Sep 11 20:00:29.098253 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091744 2546 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 11 20:00:29.098253 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091747 2546 flags.go:64] FLAG: --http-check-frequency="20s" Sep 11 20:00:29.098253 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091750 2546 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 11 20:00:29.098253 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091753 2546 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 11 20:00:29.098253 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091757 2546 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 11 20:00:29.098253 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091759 2546 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 11 20:00:29.098253 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091762 2546 flags.go:64] FLAG: --image-service-endpoint="" Sep 11 20:00:29.098253 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091765 2546 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 11 20:00:29.098253 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091768 2546 flags.go:64] FLAG: --kube-api-burst="100" Sep 11 20:00:29.098253 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091771 2546 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 11 20:00:29.098253 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091774 2546 flags.go:64] FLAG: --kube-api-qps="50" Sep 11 20:00:29.098253 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091776 2546 flags.go:64] FLAG: --kube-reserved="" Sep 11 20:00:29.098845 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091779 2546 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 11 20:00:29.098845 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091782 2546 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 11 20:00:29.098845 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091785 2546 flags.go:64] FLAG: --kubelet-cgroups="" Sep 11 20:00:29.098845 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091788 2546 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 11 20:00:29.098845 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091791 2546 flags.go:64] FLAG: --lock-file="" Sep 11 20:00:29.098845 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091795 2546 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 11 20:00:29.098845 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091798 2546 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 11 20:00:29.098845 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091801 2546 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 11 20:00:29.098845 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091807 2546 flags.go:64] FLAG: --log-json-split-stream="false" Sep 11 20:00:29.098845 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091810 2546 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 11 20:00:29.098845 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091812 2546 flags.go:64] FLAG: --log-text-split-stream="false" Sep 11 20:00:29.098845 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091815 2546 flags.go:64] FLAG: --logging-format="text" Sep 11 20:00:29.098845 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091818 2546 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 11 20:00:29.098845 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091821 2546 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 11 20:00:29.098845 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091824 2546 flags.go:64] FLAG: --manifest-url="" Sep 11 20:00:29.098845 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091827 2546 flags.go:64] FLAG: --manifest-url-header="" Sep 11 20:00:29.098845 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091831 2546 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 11 20:00:29.098845 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091834 2546 flags.go:64] FLAG: --max-open-files="1000000" Sep 11 20:00:29.098845 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091838 2546 flags.go:64] FLAG: --max-pods="110" Sep 11 20:00:29.098845 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091841 2546 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 11 20:00:29.098845 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091844 2546 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 11 20:00:29.098845 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091847 2546 flags.go:64] FLAG: --memory-manager-policy="None" Sep 11 20:00:29.098845 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091849 2546 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 11 20:00:29.098845 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091852 2546 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 11 20:00:29.098845 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091855 2546 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 11 20:00:29.099531 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091858 2546 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 11 20:00:29.099531 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091865 2546 flags.go:64] FLAG: --node-status-max-images="50" Sep 11 20:00:29.099531 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091868 2546 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 11 20:00:29.099531 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091871 2546 flags.go:64] FLAG: --oom-score-adj="-999" Sep 11 20:00:29.099531 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091874 2546 flags.go:64] FLAG: --pod-cidr="" Sep 11 20:00:29.099531 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091877 2546 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 11 20:00:29.099531 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091882 2546 flags.go:64] FLAG: --pod-manifest-path="" Sep 11 20:00:29.099531 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091885 2546 flags.go:64] FLAG: --pod-max-pids="-1" Sep 11 20:00:29.099531 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091888 2546 flags.go:64] FLAG: --pods-per-core="0" Sep 11 20:00:29.099531 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091891 2546 flags.go:64] FLAG: --port="10250" Sep 11 20:00:29.099531 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091893 2546 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 11 20:00:29.099531 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091896 2546 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-04a69c890c602c701" Sep 11 20:00:29.099531 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091900 2546 flags.go:64] FLAG: --qos-reserved="" Sep 11 20:00:29.099531 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091904 2546 flags.go:64] FLAG: --read-only-port="10255" Sep 11 20:00:29.099531 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091907 2546 flags.go:64] FLAG: --register-node="true" Sep 11 20:00:29.099531 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091910 2546 flags.go:64] FLAG: --register-with-taints="" Sep 11 20:00:29.099531 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091914 2546 flags.go:64] FLAG: --registry-burst="10" Sep 11 20:00:29.099531 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091917 2546 flags.go:64] FLAG: --registry-qps="5" Sep 11 20:00:29.099531 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091920 2546 flags.go:64] FLAG: --reserved-cpus="" Sep 11 20:00:29.099531 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091923 2546 flags.go:64] FLAG: --reserved-memory="" Sep 11 20:00:29.099531 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091926 2546 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 11 20:00:29.099531 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091929 2546 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 11 20:00:29.099531 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091932 2546 flags.go:64] FLAG: --rotate-certificates="false" Sep 11 20:00:29.099531 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091935 2546 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 11 20:00:29.099531 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091938 2546 flags.go:64] FLAG: --runonce="false" Sep 11 20:00:29.100653 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091941 2546 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 11 20:00:29.100653 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091944 2546 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 11 20:00:29.100653 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091949 2546 flags.go:64] FLAG: --seccomp-default="false" Sep 11 20:00:29.100653 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091952 2546 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 11 20:00:29.100653 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091956 2546 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 11 20:00:29.100653 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091959 2546 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 11 20:00:29.100653 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091963 2546 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 11 20:00:29.100653 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091966 2546 flags.go:64] FLAG: --storage-driver-password="root" Sep 11 20:00:29.100653 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091969 2546 flags.go:64] FLAG: --storage-driver-secure="false" Sep 11 20:00:29.100653 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091972 2546 flags.go:64] FLAG: --storage-driver-table="stats" Sep 11 20:00:29.100653 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091975 2546 flags.go:64] FLAG: --storage-driver-user="root" Sep 11 20:00:29.100653 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091978 2546 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 11 20:00:29.100653 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091981 2546 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 11 20:00:29.100653 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091984 2546 flags.go:64] FLAG: --system-cgroups="" Sep 11 20:00:29.100653 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091986 2546 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 11 20:00:29.100653 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091992 2546 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 11 20:00:29.100653 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091995 2546 flags.go:64] FLAG: --tls-cert-file="" Sep 11 20:00:29.100653 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.091997 2546 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 11 20:00:29.100653 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.092001 2546 flags.go:64] FLAG: --tls-min-version="" Sep 11 20:00:29.100653 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.092004 2546 flags.go:64] FLAG: --tls-private-key-file="" Sep 11 20:00:29.100653 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.092009 2546 flags.go:64] FLAG: --topology-manager-policy="none" Sep 11 20:00:29.100653 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.092023 2546 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 11 20:00:29.100653 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.092027 2546 flags.go:64] FLAG: --topology-manager-scope="container" Sep 11 20:00:29.100653 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.092030 2546 flags.go:64] FLAG: --v="2" Sep 11 20:00:29.100653 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.092035 2546 flags.go:64] FLAG: --version="false" Sep 11 20:00:29.101668 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.092038 2546 flags.go:64] FLAG: --vmodule="" Sep 11 20:00:29.101668 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.092042 2546 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 11 20:00:29.101668 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.092045 2546 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 11 20:00:29.101668 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092162 2546 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 20:00:29.101668 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092166 2546 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 20:00:29.101668 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092169 2546 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 20:00:29.101668 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092171 2546 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 20:00:29.101668 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092175 2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 20:00:29.101668 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092178 2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 20:00:29.101668 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092183 2546 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 20:00:29.101668 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092186 2546 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 20:00:29.101668 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092189 2546 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 20:00:29.101668 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092191 2546 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 20:00:29.101668 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092194 2546 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 20:00:29.101668 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092196 2546 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 20:00:29.101668 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092199 2546 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 20:00:29.101668 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092202 2546 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 20:00:29.101668 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092204 2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 20:00:29.101668 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092207 2546 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 20:00:29.101668 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092210 2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 20:00:29.102238 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092212 2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 20:00:29.102238 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092215 2546 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 20:00:29.102238 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092217 2546 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 20:00:29.102238 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092220 2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 20:00:29.102238 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092222 2546 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 20:00:29.102238 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092225 2546 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 20:00:29.102238 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092228 2546 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 20:00:29.102238 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092231 2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 20:00:29.102238 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092234 2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 20:00:29.102238 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092236 2546 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 20:00:29.102238 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092239 2546 feature_gate.go:328] unrecognized feature gate: Example Sep 11 20:00:29.102238 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092242 2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 20:00:29.102238 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092245 2546 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 20:00:29.102238 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092247 2546 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 20:00:29.102238 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092250 2546 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 20:00:29.102238 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092253 2546 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 20:00:29.102238 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092256 2546 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 20:00:29.102238 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092258 2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 20:00:29.102238 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092261 2546 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 20:00:29.102238 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092264 2546 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 20:00:29.102728 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092268 2546 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 20:00:29.102728 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092273 2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 20:00:29.102728 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092276 2546 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 20:00:29.102728 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092279 2546 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 20:00:29.102728 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092282 2546 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 20:00:29.102728 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092285 2546 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 20:00:29.102728 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092288 2546 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 20:00:29.102728 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092291 2546 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 20:00:29.102728 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092294 2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 20:00:29.102728 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092297 2546 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 20:00:29.102728 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092300 2546 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 20:00:29.102728 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092302 2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 20:00:29.102728 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092305 2546 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 20:00:29.102728 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092308 2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 20:00:29.102728 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092312 2546 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 20:00:29.102728 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092315 2546 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 20:00:29.102728 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092318 2546 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 20:00:29.102728 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092322 2546 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 20:00:29.102728 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092324 2546 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 20:00:29.103212 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092328 2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 20:00:29.103212 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092331 2546 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 20:00:29.103212 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092334 2546 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 20:00:29.103212 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092337 2546 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 20:00:29.103212 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092340 2546 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 20:00:29.103212 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092342 2546 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 20:00:29.103212 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092345 2546 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 20:00:29.103212 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092348 2546 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 20:00:29.103212 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092351 2546 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 20:00:29.103212 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092353 2546 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 20:00:29.103212 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092356 2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 20:00:29.103212 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092358 2546 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 20:00:29.103212 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092361 2546 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 20:00:29.103212 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092363 2546 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 20:00:29.103212 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092367 2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 20:00:29.103212 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092370 2546 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 20:00:29.103212 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092372 2546 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 20:00:29.103212 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092375 2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 20:00:29.103212 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092377 2546 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 20:00:29.103212 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092380 2546 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 20:00:29.103719 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092384 2546 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 20:00:29.103719 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092387 2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 20:00:29.103719 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092390 2546 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 20:00:29.103719 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092392 2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 20:00:29.103719 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092395 2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 20:00:29.103719 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092398 2546 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 20:00:29.103719 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092400 2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 20:00:29.103719 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092403 2546 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 20:00:29.103719 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092405 2546 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 20:00:29.103719 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092408 2546 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 20:00:29.103719 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092411 2546 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 20:00:29.103719 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092413 2546 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 20:00:29.103719 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092417 2546 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 20:00:29.103719 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092420 2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 20:00:29.103719 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092423 2546 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 20:00:29.103719 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092425 2546 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 20:00:29.103719 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092428 2546 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 20:00:29.103719 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092431 2546 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 20:00:29.103719 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092433 2546 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 20:00:29.104249 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092436 2546 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 20:00:29.104249 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092439 2546 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 20:00:29.104249 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092441 2546 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 20:00:29.104249 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.092445 2546 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 20:00:29.104249 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.093167 2546 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 20:00:29.104249 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.102280 2546 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 11 20:00:29.104249 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.102295 2546 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 11 20:00:29.104249 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.103225 2546 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 11 20:00:29.104249 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.103233 2546 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 11 20:00:29.104249 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103289 2546 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 20:00:29.104249 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103294 2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 20:00:29.104249 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103298 2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 20:00:29.104249 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103302 2546 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 20:00:29.104249 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103305 2546 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 20:00:29.104249 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103308 2546 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 20:00:29.104630 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103310 2546 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 20:00:29.104630 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103313 2546 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 20:00:29.104630 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103316 2546 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 20:00:29.104630 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103319 2546 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 20:00:29.104630 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103321 2546 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 20:00:29.104630 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103324 2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 20:00:29.104630 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103327 2546 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 20:00:29.104630 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103330 2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 20:00:29.104630 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103332 2546 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 20:00:29.104630 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103335 2546 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 20:00:29.104630 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103337 2546 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 20:00:29.104630 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103340 2546 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 20:00:29.104630 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103344 2546 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 20:00:29.104630 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103347 2546 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 20:00:29.104630 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103349 2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 20:00:29.104630 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103352 2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 20:00:29.104630 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103354 2546 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 20:00:29.104630 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103357 2546 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 20:00:29.104630 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103360 2546 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 20:00:29.104630 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103362 2546 feature_gate.go:328] unrecognized feature gate: Example Sep 11 20:00:29.105135 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103365 2546 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 20:00:29.105135 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103368 2546 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 20:00:29.105135 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103371 2546 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 20:00:29.105135 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103374 2546 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 20:00:29.105135 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103377 2546 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 20:00:29.105135 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103380 2546 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 20:00:29.105135 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103382 2546 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 20:00:29.105135 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103385 2546 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 20:00:29.105135 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103388 2546 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 20:00:29.105135 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103390 2546 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 20:00:29.105135 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103393 2546 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 20:00:29.105135 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103395 2546 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 20:00:29.105135 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103398 2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 20:00:29.105135 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103401 2546 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 20:00:29.105135 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103403 2546 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 20:00:29.105135 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103408 2546 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 20:00:29.105135 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103411 2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 20:00:29.105135 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103414 2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 20:00:29.105135 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103417 2546 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 20:00:29.105608 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103420 2546 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 20:00:29.105608 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103422 2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 20:00:29.105608 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103425 2546 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 20:00:29.105608 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103427 2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 20:00:29.105608 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103430 2546 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 20:00:29.105608 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103433 2546 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 20:00:29.105608 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103436 2546 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 20:00:29.105608 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103439 2546 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 20:00:29.105608 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103441 2546 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 20:00:29.105608 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103444 2546 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 20:00:29.105608 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103447 2546 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 20:00:29.105608 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103449 2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 20:00:29.105608 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103452 2546 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 20:00:29.105608 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103455 2546 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 20:00:29.105608 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103458 2546 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 20:00:29.105608 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103460 2546 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 20:00:29.105608 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103463 2546 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 20:00:29.105608 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103466 2546 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 20:00:29.105608 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103469 2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 20:00:29.105608 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103471 2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 20:00:29.106120 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103475 2546 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 20:00:29.106120 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103477 2546 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 20:00:29.106120 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103480 2546 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 20:00:29.106120 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103483 2546 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 20:00:29.106120 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103485 2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 20:00:29.106120 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103488 2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 20:00:29.106120 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103490 2546 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 20:00:29.106120 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103494 2546 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 20:00:29.106120 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103498 2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 20:00:29.106120 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103501 2546 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 20:00:29.106120 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103504 2546 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 20:00:29.106120 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103507 2546 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 20:00:29.106120 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103509 2546 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 20:00:29.106120 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103512 2546 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 20:00:29.106120 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103516 2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 20:00:29.106120 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103520 2546 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 20:00:29.106120 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103523 2546 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 20:00:29.106120 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103527 2546 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 20:00:29.106120 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103529 2546 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 20:00:29.106588 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103532 2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 20:00:29.106588 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103535 2546 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 20:00:29.106588 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103538 2546 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 20:00:29.106588 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103541 2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 20:00:29.106588 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103544 2546 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 20:00:29.106588 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103548 2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 20:00:29.106588 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103551 2546 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 20:00:29.106588 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103554 2546 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 20:00:29.106588 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103557 2546 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 20:00:29.106588 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103561 2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 20:00:29.106588 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103564 2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 20:00:29.106588 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103567 2546 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 20:00:29.106588 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103570 2546 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 20:00:29.106588 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103572 2546 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 20:00:29.106588 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103575 2546 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 20:00:29.106958 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.103580 2546 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 20:00:29.106958 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103696 2546 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 20:00:29.106958 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103701 2546 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 20:00:29.106958 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103705 2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 20:00:29.106958 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103708 2546 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 20:00:29.106958 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103711 2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 20:00:29.106958 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103713 2546 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 20:00:29.106958 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103716 2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 20:00:29.106958 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103720 2546 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 20:00:29.106958 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103724 2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 20:00:29.106958 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103727 2546 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 20:00:29.106958 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103730 2546 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 20:00:29.106958 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103734 2546 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 20:00:29.106958 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103737 2546 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 20:00:29.106958 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103740 2546 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 20:00:29.107339 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103743 2546 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 20:00:29.107339 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103746 2546 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 20:00:29.107339 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103749 2546 feature_gate.go:328] unrecognized feature gate: Example Sep 11 20:00:29.107339 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103753 2546 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 20:00:29.107339 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103756 2546 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 20:00:29.107339 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103758 2546 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 20:00:29.107339 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103761 2546 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 20:00:29.107339 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103764 2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 20:00:29.107339 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103766 2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 20:00:29.107339 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103769 2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 20:00:29.107339 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103771 2546 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 20:00:29.107339 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103774 2546 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 20:00:29.107339 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103777 2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 20:00:29.107339 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103779 2546 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 20:00:29.107339 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103782 2546 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 20:00:29.107339 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103784 2546 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 20:00:29.107339 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103787 2546 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 20:00:29.107339 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103789 2546 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 20:00:29.107339 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103792 2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 20:00:29.107339 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103795 2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 20:00:29.107835 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103797 2546 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 20:00:29.107835 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103800 2546 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 20:00:29.107835 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103802 2546 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 20:00:29.107835 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103805 2546 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 20:00:29.107835 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103808 2546 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 20:00:29.107835 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103812 2546 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 20:00:29.107835 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103815 2546 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 20:00:29.107835 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103817 2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 20:00:29.107835 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103820 2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 20:00:29.107835 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103822 2546 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 20:00:29.107835 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103825 2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 20:00:29.107835 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103828 2546 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 20:00:29.107835 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103831 2546 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 20:00:29.107835 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103834 2546 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 20:00:29.107835 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103836 2546 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 20:00:29.107835 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103839 2546 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 20:00:29.107835 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103841 2546 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 20:00:29.107835 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103844 2546 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 20:00:29.107835 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103847 2546 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 20:00:29.108316 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103849 2546 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 20:00:29.108316 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103852 2546 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 20:00:29.108316 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103854 2546 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 20:00:29.108316 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103857 2546 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 20:00:29.108316 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103860 2546 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 20:00:29.108316 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103862 2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 20:00:29.108316 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103865 2546 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 20:00:29.108316 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103868 2546 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 20:00:29.108316 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103870 2546 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 20:00:29.108316 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103873 2546 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 20:00:29.108316 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103875 2546 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 20:00:29.108316 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103878 2546 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 20:00:29.108316 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103880 2546 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 20:00:29.108316 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103883 2546 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 20:00:29.108316 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103885 2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 20:00:29.108316 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103888 2546 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 20:00:29.108316 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103890 2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 20:00:29.108316 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103893 2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 20:00:29.108316 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103895 2546 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 20:00:29.108316 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103898 2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 20:00:29.108810 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103900 2546 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 20:00:29.108810 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103903 2546 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 20:00:29.108810 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103905 2546 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 20:00:29.108810 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103908 2546 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 20:00:29.108810 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103910 2546 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 20:00:29.108810 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103914 2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 20:00:29.108810 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103917 2546 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 20:00:29.108810 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103919 2546 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 20:00:29.108810 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103922 2546 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 20:00:29.108810 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103924 2546 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 20:00:29.108810 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103927 2546 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 20:00:29.108810 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103930 2546 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 20:00:29.108810 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103932 2546 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 20:00:29.108810 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103935 2546 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 20:00:29.108810 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103937 2546 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 20:00:29.108810 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103940 2546 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 20:00:29.108810 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103942 2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 20:00:29.108810 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103945 2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 20:00:29.108810 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103948 2546 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 20:00:29.108810 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103950 2546 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 20:00:29.109311 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103953 2546 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 20:00:29.109311 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103955 2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 20:00:29.109311 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103958 2546 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 20:00:29.109311 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103962 2546 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 20:00:29.109311 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103965 2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 20:00:29.109311 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:29.103968 2546 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 20:00:29.109311 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.103973 2546 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 20:00:29.109311 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.104090 2546 server.go:962] "Client rotation is on, will bootstrap in background" Sep 11 20:00:29.109519 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.109451 2546 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 11 20:00:29.110572 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.110560 2546 server.go:1019] "Starting client certificate rotation" Sep 11 20:00:29.110672 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.110655 2546 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 11 20:00:29.110702 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.110687 2546 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 11 20:00:29.113529 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.113516 2546 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 11 20:00:29.114676 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.114658 2546 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 11 20:00:29.125387 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.125369 2546 log.go:25] "Validated CRI v1 runtime API" Sep 11 20:00:29.129041 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.129011 2546 log.go:25] "Validated CRI v1 image API" Sep 11 20:00:29.130965 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.130948 2546 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 11 20:00:29.133596 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.133576 2546 fs.go:135] Filesystem UUIDs: map[2a87b5f2-322b-4e91-a3be-759135ac5d00:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d031a70b-fa21-482e-9e97-c99a5cab6ed0:/dev/nvme0n1p3] Sep 11 20:00:29.133669 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.133594 2546 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 11 20:00:29.135848 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.135829 2546 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 11 20:00:29.137815 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.137708 2546 manager.go:216] Machine: {Timestamp:2026-09-11 20:00:29.137252425 +0000 UTC m=+0.363862827 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3115121 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f23af0457351e5990b6417fbcebe3 SystemUUID:ec2f23af-0457-351e-5990-b6417fbcebe3 BootID:ea4b23c5-846a-42b3-b1b6-a3e274472ce6 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:dc:f0:9f:c2:6b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:dc:f0:9f:c2:6b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c6:f5:c0:59:0a:31 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 11 20:00:29.137815 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.137802 2546 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 11 20:00:29.137961 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.137895 2546 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 11 20:00:29.141596 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.141574 2546 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 11 20:00:29.141725 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.141599 2546 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-205.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 11 20:00:29.143088 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.143077 2546 topology_manager.go:138] "Creating topology manager with none policy" Sep 11 20:00:29.143121 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.143090 2546 container_manager_linux.go:309] "Creating device plugin manager" Sep 11 20:00:29.143121 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.143098 2546 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 11 20:00:29.144122 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.144112 2546 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 11 20:00:29.145309 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.145299 2546 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 11 20:00:29.150367 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.150355 2546 state_mem.go:36] "Initialized new in-memory state store" Sep 11 20:00:29.150488 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.150479 2546 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 11 20:00:29.150522 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.150508 2546 kubelet.go:486] "Attempting to sync node with API server" Sep 11 20:00:29.150565 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.150525 2546 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 11 20:00:29.150565 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.150543 2546 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 11 20:00:29.150565 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.150552 2546 kubelet.go:398] "Adding apiserver pod source" Sep 11 20:00:29.150565 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.150561 2546 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 11 20:00:29.151750 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.151731 2546 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 11 20:00:29.151826 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.151763 2546 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 11 20:00:29.154766 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.154745 2546 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 11 20:00:29.157239 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.157225 2546 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 11 20:00:29.157311 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.157256 2546 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 11 20:00:29.160046 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.160031 2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 11 20:00:29.160046 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.160047 2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 11 20:00:29.160179 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.160053 2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 11 20:00:29.160179 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.160058 2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 11 20:00:29.160179 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.160067 2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 11 20:00:29.160179 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.160072 2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 11 20:00:29.160179 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.160086 2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 11 20:00:29.160179 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.160091 2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 11 20:00:29.160179 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.160098 2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 11 20:00:29.160179 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.160103 2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 11 20:00:29.160179 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.160111 2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 11 20:00:29.160179 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.160116 2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 11 20:00:29.161032 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.161001 2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 11 20:00:29.161032 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.161032 2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 11 20:00:29.161167 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:29.161146 2546 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-205.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 11 20:00:29.161216 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.161170 2546 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bt96m" Sep 11 20:00:29.161216 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:29.161200 2546 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 11 20:00:29.164599 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.164587 2546 server.go:1268] "Started kubelet" Sep 11 20:00:29.164701 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.164674 2546 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 11 20:00:29.164814 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.164665 2546 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 11 20:00:29.164917 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.164832 2546 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 11 20:00:29.165452 ip-10-0-138-205 systemd[1]: Started Kubernetes Kubelet. Sep 11 20:00:29.166511 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.166493 2546 server.go:310] "Adding debug handlers to kubelet server" Sep 11 20:00:29.168968 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:29.168949 2546 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 11 20:00:29.169078 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.169066 2546 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 11 20:00:29.170062 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.170043 2546 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bt96m" Sep 11 20:00:29.172132 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.172115 2546 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-205.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 11 20:00:29.173179 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:29.171979 2546 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-205.ec2.internal.18d45ca821d163b7 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-205.ec2.internal,UID:ip-10-0-138-205.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-205.ec2.internal,},FirstTimestamp:2026-09-11 20:00:29.164561335 +0000 UTC m=+0.391171737,LastTimestamp:2026-09-11 20:00:29.164561335 +0000 UTC m=+0.391171737,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-205.ec2.internal,}" Sep 11 20:00:29.176210 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.176191 2546 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 11 20:00:29.176903 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.176886 2546 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 11 20:00:29.177117 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:29.177075 2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-205.ec2.internal\" not found" Sep 11 20:00:29.177750 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.177594 2546 factory.go:55] Registering systemd factory Sep 11 20:00:29.177819 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.177771 2546 factory.go:223] Registration of the systemd container factory successfully Sep 11 20:00:29.177819 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.177692 2546 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 11 20:00:29.177819 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.177691 2546 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 11 20:00:29.177819 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.177819 2546 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 11 20:00:29.177980 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.177906 2546 reconstruct.go:97] "Volume reconstruction finished" Sep 11 20:00:29.177980 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.177916 2546 reconciler.go:29] "Reconciler: start to sync state" Sep 11 20:00:29.177980 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.177978 2546 factory.go:153] Registering CRI-O factory Sep 11 20:00:29.178100 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.177988 2546 factory.go:223] Registration of the crio container factory successfully Sep 11 20:00:29.178100 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.178054 2546 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 11 20:00:29.178100 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.178078 2546 factory.go:103] Registering Raw factory Sep 11 20:00:29.178100 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.178090 2546 manager.go:1196] Started watching for new ooms in manager Sep 11 20:00:29.178443 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.178431 2546 manager.go:349] Starting recovery of all containers Sep 11 20:00:29.179229 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.179208 2546 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 20:00:29.184153 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:29.184128 2546 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-138-205.ec2.internal\" not found" node="ip-10-0-138-205.ec2.internal" Sep 11 20:00:29.189939 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.189922 2546 manager.go:354] Recovery completed Sep 11 20:00:29.194003 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.193991 2546 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 20:00:29.195955 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.195939 2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-205.ec2.internal" event="NodeHasSufficientMemory" Sep 11 20:00:29.196029 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.195965 2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-205.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 20:00:29.196029 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.195975 2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-205.ec2.internal" event="NodeHasSufficientPID" Sep 11 20:00:29.196470 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.196457 2546 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 11 20:00:29.196470 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.196470 2546 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 20:00:29.196568 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.196469 2546 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 11 20:00:29.196568 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.196542 2546 state_mem.go:36] "Initialized new in-memory state store" Sep 11 20:00:29.197481 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.197464 2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-205.ec2.internal" event="NodeHasSufficientMemory" Sep 11 20:00:29.197534 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.197494 2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-205.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 20:00:29.197534 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.197505 2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-205.ec2.internal" event="NodeHasSufficientPID" Sep 11 20:00:29.198662 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.198648 2546 policy_none.go:49] "None policy: Start" Sep 11 20:00:29.198698 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.198668 2546 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 11 20:00:29.198698 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.198682 2546 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 11 20:00:29.200692 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.200674 2546 policy_none.go:47] "Start" Sep 11 20:00:29.227803 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.227786 2546 manager.go:337] "Starting Device Plugin manager" Sep 11 20:00:29.253974 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:29.227831 2546 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 11 20:00:29.253974 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.227845 2546 server.go:87] "Starting device plugin registration server" Sep 11 20:00:29.253974 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.228208 2546 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 11 20:00:29.253974 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.228221 2546 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 11 20:00:29.253974 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.228332 2546 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 11 20:00:29.253974 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.228767 2546 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 11 20:00:29.253974 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.228778 2546 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 11 20:00:29.253974 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:29.228855 2546 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 11 20:00:29.253974 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:29.228912 2546 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-205.ec2.internal\" not found" Sep 11 20:00:29.263669 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.263647 2546 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 11 20:00:29.264798 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.264779 2546 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 11 20:00:29.264871 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.264831 2546 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 11 20:00:29.264871 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.264854 2546 kubelet.go:2448] "Starting kubelet main sync loop" Sep 11 20:00:29.264992 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:29.264893 2546 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 11 20:00:29.267447 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.267426 2546 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 20:00:29.328896 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.328871 2546 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 20:00:29.329749 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.329734 2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-205.ec2.internal" event="NodeHasSufficientMemory" Sep 11 20:00:29.329819 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.329763 2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-205.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 20:00:29.329819 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.329776 2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-205.ec2.internal" event="NodeHasSufficientPID" Sep 11 20:00:29.329819 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.329802 2546 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-205.ec2.internal" Sep 11 20:00:29.335926 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.335855 2546 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-205.ec2.internal" Sep 11 20:00:29.335926 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:29.335877 2546 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-205.ec2.internal\": node \"ip-10-0-138-205.ec2.internal\" not found" Sep 11 20:00:29.354612 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:29.354590 2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-205.ec2.internal\" not found" Sep 11 20:00:29.365444 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.365423 2546 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-138-205.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-205.ec2.internal"] Sep 11 20:00:29.365502 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.365483 2546 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 20:00:29.366796 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.366782 2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-205.ec2.internal" event="NodeHasSufficientMemory" Sep 11 20:00:29.366862 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.366809 2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-205.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 20:00:29.366862 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.366819 2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-205.ec2.internal" event="NodeHasSufficientPID" Sep 11 20:00:29.369137 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.369125 2546 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 20:00:29.369261 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.369247 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-205.ec2.internal" Sep 11 20:00:29.369297 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.369276 2546 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 20:00:29.369858 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.369844 2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-205.ec2.internal" event="NodeHasSufficientMemory" Sep 11 20:00:29.369929 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.369868 2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-205.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 20:00:29.369929 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.369882 2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-205.ec2.internal" event="NodeHasSufficientPID" Sep 11 20:00:29.369929 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.369891 2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-205.ec2.internal" event="NodeHasSufficientMemory" Sep 11 20:00:29.369929 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.369908 2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-205.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 20:00:29.369929 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.369917 2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-205.ec2.internal" event="NodeHasSufficientPID" Sep 11 20:00:29.372479 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.372463 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-205.ec2.internal" Sep 11 20:00:29.372558 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.372490 2546 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 20:00:29.373150 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.373130 2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-205.ec2.internal" event="NodeHasSufficientMemory" Sep 11 20:00:29.373217 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.373158 2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-205.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 20:00:29.373217 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.373171 2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-205.ec2.internal" event="NodeHasSufficientPID" Sep 11 20:00:29.379635 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.379619 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/11ecaef2c363c5ae25ba196bd5dc1142-config\") pod \"kube-apiserver-proxy-ip-10-0-138-205.ec2.internal\" (UID: \"11ecaef2c363c5ae25ba196bd5dc1142\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-205.ec2.internal" Sep 11 20:00:29.379721 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.379640 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d907173ce62f903e18094a4e7845248d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-205.ec2.internal\" (UID: \"d907173ce62f903e18094a4e7845248d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-205.ec2.internal" Sep 11 20:00:29.379721 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.379658 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d907173ce62f903e18094a4e7845248d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-205.ec2.internal\" (UID: \"d907173ce62f903e18094a4e7845248d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-205.ec2.internal" Sep 11 20:00:29.397574 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:29.397553 2546 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-205.ec2.internal\" not found" node="ip-10-0-138-205.ec2.internal" Sep 11 20:00:29.401977 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:29.401963 2546 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-205.ec2.internal\" not found" node="ip-10-0-138-205.ec2.internal" Sep 11 20:00:29.454814 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:29.454793 2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-205.ec2.internal\" not found" Sep 11 20:00:29.480107 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.480085 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/11ecaef2c363c5ae25ba196bd5dc1142-config\") pod \"kube-apiserver-proxy-ip-10-0-138-205.ec2.internal\" (UID: \"11ecaef2c363c5ae25ba196bd5dc1142\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-205.ec2.internal" Sep 11 20:00:29.480194 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.480111 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d907173ce62f903e18094a4e7845248d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-205.ec2.internal\" (UID: \"d907173ce62f903e18094a4e7845248d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-205.ec2.internal" Sep 11 20:00:29.480194 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.480128 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d907173ce62f903e18094a4e7845248d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-205.ec2.internal\" (UID: \"d907173ce62f903e18094a4e7845248d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-205.ec2.internal" Sep 11 20:00:29.480194 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.480155 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d907173ce62f903e18094a4e7845248d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-205.ec2.internal\" (UID: \"d907173ce62f903e18094a4e7845248d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-205.ec2.internal" Sep 11 20:00:29.480194 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.480172 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/11ecaef2c363c5ae25ba196bd5dc1142-config\") pod \"kube-apiserver-proxy-ip-10-0-138-205.ec2.internal\" (UID: \"11ecaef2c363c5ae25ba196bd5dc1142\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-205.ec2.internal" Sep 11 20:00:29.480194 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.480184 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d907173ce62f903e18094a4e7845248d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-205.ec2.internal\" (UID: \"d907173ce62f903e18094a4e7845248d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-205.ec2.internal" Sep 11 20:00:29.555151 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:29.555114 2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-205.ec2.internal\" not found" Sep 11 20:00:29.655923 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:29.655842 2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-205.ec2.internal\" not found" Sep 11 20:00:29.700314 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.700292 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-205.ec2.internal" Sep 11 20:00:29.705357 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:29.705335 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-205.ec2.internal" Sep 11 20:00:29.756011 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:29.755972 2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-205.ec2.internal\" not found" Sep 11 20:00:29.856477 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:29.856449 2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-205.ec2.internal\" not found" Sep 11 20:00:31.135622 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:31.135534 2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-205.ec2.internal\" not found" Sep 11 20:00:31.236015 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:31.235982 2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-205.ec2.internal\" not found" Sep 11 20:00:31.289505 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:31.289483 2546 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 11 20:00:31.289878 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:31.289633 2546 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 11 20:00:31.289878 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:31.289632 2546 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 11 20:00:31.336990 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:31.336968 2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-205.ec2.internal\" not found" Sep 11 20:00:31.354940 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:31.354907 2546 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 19:55:30 +0000 UTC" deadline="2028-03-04 23:25:24.546359591 +0000 UTC" Sep 11 20:00:31.354991 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:31.354939 2546 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12963h24m53.19142384s" Sep 11 20:00:31.437315 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:31.437264 2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-205.ec2.internal\" not found" Sep 11 20:00:31.537685 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:31.537647 2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-205.ec2.internal\" not found" Sep 11 20:00:31.638287 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:31.638262 2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-205.ec2.internal\" not found" Sep 11 20:00:31.643323 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:31.643299 2546 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 20:00:31.726227 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:31.726195 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11ecaef2c363c5ae25ba196bd5dc1142.slice/crio-fbae09b8a86fa2ba561ba9fc744d4359825a104e4c5be3611180c22e212d12d1 WatchSource:0}: Error finding container fbae09b8a86fa2ba561ba9fc744d4359825a104e4c5be3611180c22e212d12d1: Status 404 returned error can't find the container with id fbae09b8a86fa2ba561ba9fc744d4359825a104e4c5be3611180c22e212d12d1 Sep 11 20:00:31.730427 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:31.730411 2546 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 20:00:31.739295 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:31.739277 2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-205.ec2.internal\" not found" Sep 11 20:00:31.840041 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:31.840008 2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-205.ec2.internal\" not found" Sep 11 20:00:31.932364 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:31.932341 2546 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 20:00:31.956022 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:31.956002 2546 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-205.ec2.internal" Sep 11 20:00:31.965492 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:31.965469 2546 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 11 20:00:31.966650 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:31.966606 2546 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-205.ec2.internal" Sep 11 20:00:31.975429 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:31.975414 2546 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 11 20:00:32.051708 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:32.051687 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd907173ce62f903e18094a4e7845248d.slice/crio-2d1daa9f8f25a8cd807dac33771ed42f2f8b353ad6b496d335f9ce5c15ca3f55 WatchSource:0}: Error finding container 2d1daa9f8f25a8cd807dac33771ed42f2f8b353ad6b496d335f9ce5c15ca3f55: Status 404 returned error can't find the container with id 2d1daa9f8f25a8cd807dac33771ed42f2f8b353ad6b496d335f9ce5c15ca3f55 Sep 11 20:00:32.330791 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.330719 2546 apiserver.go:52] "Watching apiserver" Sep 11 20:00:32.339771 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.339747 2546 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 11 20:00:32.340181 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.340154 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-205.ec2.internal","openshift-multus/network-metrics-daemon-zpl5q","kube-system/global-pull-secret-syncer-q2dm2","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w6p5s","openshift-multus/multus-8kcqg","openshift-multus/multus-additional-cni-plugins-ddmz9","openshift-network-diagnostics/network-check-target-wk9hk","openshift-network-operator/iptables-alerter-r64rv","openshift-ovn-kubernetes/ovnkube-node-pgnnf","kube-system/konnectivity-agent-jdhk4","kube-system/kube-apiserver-proxy-ip-10-0-138-205.ec2.internal","openshift-cluster-node-tuning-operator/tuned-f6skx","openshift-dns/node-resolver-2m6cs","openshift-image-registry/node-ca-54fff"] Sep 11 20:00:32.343338 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.343315 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wk9hk" Sep 11 20:00:32.343443 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:32.343399 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wk9hk" podUID="ae2b5d1f-4bb1-41eb-9096-d54b304ad672" Sep 11 20:00:32.345683 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.345660 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2m6cs" Sep 11 20:00:32.347630 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.347612 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 11 20:00:32.347732 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.347645 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tsbfl\"" Sep 11 20:00:32.347833 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.347816 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 11 20:00:32.348506 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.348410 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2dm2" Sep 11 20:00:32.348506 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:32.348476 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2dm2" podUID="1f86594f-112f-4e58-a2ca-bbd951c4d1af" Sep 11 20:00:32.350697 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.350675 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w6p5s" Sep 11 20:00:32.352897 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.352732 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-z4fwg\"" Sep 11 20:00:32.352897 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.352750 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 11 20:00:32.352897 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.352736 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 11 20:00:32.352897 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.352808 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 11 20:00:32.353127 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.353031 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8kcqg" Sep 11 20:00:32.354741 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.354719 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 11 20:00:32.354986 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.354945 2546 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 11 20:00:32.355071 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.355045 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 11 20:00:32.355133 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.355088 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-f6rpl\"" Sep 11 20:00:32.355236 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.355185 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 11 20:00:32.355387 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.355365 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 11 20:00:32.356016 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.355995 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ddmz9" Sep 11 20:00:32.357855 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.357818 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 11 20:00:32.357855 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.357820 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 11 20:00:32.357990 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.357837 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5b95w\"" Sep 11 20:00:32.358608 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.358587 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:32.360326 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.360309 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 11 20:00:32.360552 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.360503 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 11 20:00:32.360758 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.360616 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 11 20:00:32.360880 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.360760 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 11 20:00:32.360880 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.360779 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5wxb6\"" Sep 11 20:00:32.361095 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.360900 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 11 20:00:32.361095 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.361015 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-54fff" Sep 11 20:00:32.361405 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.361385 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 11 20:00:32.363135 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.362902 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 11 20:00:32.363135 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.363038 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jdjsc\"" Sep 11 20:00:32.363135 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.363113 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zpl5q" Sep 11 20:00:32.363346 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:32.363172 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zpl5q" podUID="ce96d552-27b5-4912-86b0-38f0f46d0002" Sep 11 20:00:32.363346 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.363292 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 11 20:00:32.363458 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.363447 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 11 20:00:32.365539 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.365515 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jdhk4" Sep 11 20:00:32.366384 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.366362 2546 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 11 20:00:32.367667 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.367648 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-xrg7n\"" Sep 11 20:00:32.367802 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.367753 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2pgj\" (UniqueName: \"kubernetes.io/projected/52601b9c-beef-4b1c-aaf3-34cf267a00fe-kube-api-access-h2pgj\") pod \"aws-ebs-csi-driver-node-w6p5s\" (UID: \"52601b9c-beef-4b1c-aaf3-34cf267a00fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w6p5s" Sep 11 20:00:32.367858 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.367825 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/171c51ac-8f0e-426f-973d-5a596e73d7d1-host-var-lib-cni-bin\") pod \"multus-8kcqg\" (UID: \"171c51ac-8f0e-426f-973d-5a596e73d7d1\") " pod="openshift-multus/multus-8kcqg" Sep 11 20:00:32.367909 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.367859 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88sjc\" (UniqueName: \"kubernetes.io/projected/171c51ac-8f0e-426f-973d-5a596e73d7d1-kube-api-access-88sjc\") pod \"multus-8kcqg\" (UID: \"171c51ac-8f0e-426f-973d-5a596e73d7d1\") " pod="openshift-multus/multus-8kcqg" Sep 11 20:00:32.367958 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.367913 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2133a2c0-f263-45aa-9d96-1819721a8c9b-hosts-file\") pod \"node-resolver-2m6cs\" (UID: \"2133a2c0-f263-45aa-9d96-1819721a8c9b\") " pod="openshift-dns/node-resolver-2m6cs" Sep 11 20:00:32.368007 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.367953 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/171c51ac-8f0e-426f-973d-5a596e73d7d1-system-cni-dir\") pod \"multus-8kcqg\" (UID: \"171c51ac-8f0e-426f-973d-5a596e73d7d1\") " pod="openshift-multus/multus-8kcqg" Sep 11 20:00:32.368007 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.367991 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/171c51ac-8f0e-426f-973d-5a596e73d7d1-multus-cni-dir\") pod \"multus-8kcqg\" (UID: \"171c51ac-8f0e-426f-973d-5a596e73d7d1\") " pod="openshift-multus/multus-8kcqg" Sep 11 20:00:32.368101 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.368018 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 11 20:00:32.368101 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.368067 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 11 20:00:32.368101 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.368020 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/171c51ac-8f0e-426f-973d-5a596e73d7d1-host-var-lib-kubelet\") pod \"multus-8kcqg\" (UID: \"171c51ac-8f0e-426f-973d-5a596e73d7d1\") " pod="openshift-multus/multus-8kcqg" Sep 11 20:00:32.368235 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.368109 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/171c51ac-8f0e-426f-973d-5a596e73d7d1-multus-daemon-config\") pod \"multus-8kcqg\" (UID: \"171c51ac-8f0e-426f-973d-5a596e73d7d1\") " pod="openshift-multus/multus-8kcqg" Sep 11 20:00:32.368235 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.368133 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b6b30385-b0e1-4ab5-916a-57992046f196-ovn-node-metrics-cert\") pod \"ovnkube-node-pgnnf\" (UID: \"b6b30385-b0e1-4ab5-916a-57992046f196\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:32.368235 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.368162 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b6b30385-b0e1-4ab5-916a-57992046f196-host-cni-netd\") pod \"ovnkube-node-pgnnf\" (UID: \"b6b30385-b0e1-4ab5-916a-57992046f196\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:32.368235 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.368189 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/171c51ac-8f0e-426f-973d-5a596e73d7d1-host-var-lib-cni-multus\") pod \"multus-8kcqg\" (UID: \"171c51ac-8f0e-426f-973d-5a596e73d7d1\") " pod="openshift-multus/multus-8kcqg" Sep 11 20:00:32.368235 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.368217 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/171c51ac-8f0e-426f-973d-5a596e73d7d1-hostroot\") pod \"multus-8kcqg\" (UID: \"171c51ac-8f0e-426f-973d-5a596e73d7d1\") " pod="openshift-multus/multus-8kcqg" Sep 11 20:00:32.368442 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.368255 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd9x6\" (UniqueName: \"kubernetes.io/projected/b6b30385-b0e1-4ab5-916a-57992046f196-kube-api-access-gd9x6\") pod \"ovnkube-node-pgnnf\" (UID: \"b6b30385-b0e1-4ab5-916a-57992046f196\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:32.368442 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.368285 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/52601b9c-beef-4b1c-aaf3-34cf267a00fe-kubelet-dir\") pod \"aws-ebs-csi-driver-node-w6p5s\" (UID: \"52601b9c-beef-4b1c-aaf3-34cf267a00fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w6p5s" Sep 11 20:00:32.368442 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.368327 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/171c51ac-8f0e-426f-973d-5a596e73d7d1-cnibin\") pod \"multus-8kcqg\" (UID: \"171c51ac-8f0e-426f-973d-5a596e73d7d1\") " pod="openshift-multus/multus-8kcqg" Sep 11 20:00:32.368442 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.368385 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/171c51ac-8f0e-426f-973d-5a596e73d7d1-host-run-multus-certs\") pod \"multus-8kcqg\" (UID: \"171c51ac-8f0e-426f-973d-5a596e73d7d1\") " pod="openshift-multus/multus-8kcqg" Sep 11 20:00:32.368442 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.368420 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/171c51ac-8f0e-426f-973d-5a596e73d7d1-cni-binary-copy\") pod \"multus-8kcqg\" (UID: \"171c51ac-8f0e-426f-973d-5a596e73d7d1\") " pod="openshift-multus/multus-8kcqg" Sep 11 20:00:32.368673 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.368465 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/171c51ac-8f0e-426f-973d-5a596e73d7d1-multus-conf-dir\") pod \"multus-8kcqg\" (UID: \"171c51ac-8f0e-426f-973d-5a596e73d7d1\") " pod="openshift-multus/multus-8kcqg" Sep 11 20:00:32.368673 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.368502 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b6b30385-b0e1-4ab5-916a-57992046f196-systemd-units\") pod \"ovnkube-node-pgnnf\" (UID: \"b6b30385-b0e1-4ab5-916a-57992046f196\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:32.368673 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.368543 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b6b30385-b0e1-4ab5-916a-57992046f196-var-lib-openvswitch\") pod \"ovnkube-node-pgnnf\" (UID: \"b6b30385-b0e1-4ab5-916a-57992046f196\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:32.368673 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.368570 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b6b30385-b0e1-4ab5-916a-57992046f196-ovnkube-config\") pod \"ovnkube-node-pgnnf\" (UID: \"b6b30385-b0e1-4ab5-916a-57992046f196\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:32.368673 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.368614 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/171c51ac-8f0e-426f-973d-5a596e73d7d1-host-run-netns\") pod \"multus-8kcqg\" (UID: \"171c51ac-8f0e-426f-973d-5a596e73d7d1\") " pod="openshift-multus/multus-8kcqg" Sep 11 20:00:32.368673 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.368653 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/171c51ac-8f0e-426f-973d-5a596e73d7d1-etc-kubernetes\") pod \"multus-8kcqg\" (UID: \"171c51ac-8f0e-426f-973d-5a596e73d7d1\") " pod="openshift-multus/multus-8kcqg" Sep 11 20:00:32.368923 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.368707 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b6b30385-b0e1-4ab5-916a-57992046f196-env-overrides\") pod \"ovnkube-node-pgnnf\" (UID: \"b6b30385-b0e1-4ab5-916a-57992046f196\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:32.368923 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.368731 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1f86594f-112f-4e58-a2ca-bbd951c4d1af-dbus\") pod \"global-pull-secret-syncer-q2dm2\" (UID: \"1f86594f-112f-4e58-a2ca-bbd951c4d1af\") " pod="kube-system/global-pull-secret-syncer-q2dm2" Sep 11 20:00:32.368923 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.368761 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/52c72e1a-d061-4d3c-922b-b572a6496fed-system-cni-dir\") pod \"multus-additional-cni-plugins-ddmz9\" (UID: \"52c72e1a-d061-4d3c-922b-b572a6496fed\") " pod="openshift-multus/multus-additional-cni-plugins-ddmz9" Sep 11 20:00:32.368923 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.368788 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6b30385-b0e1-4ab5-916a-57992046f196-host-kubelet\") pod \"ovnkube-node-pgnnf\" (UID: \"b6b30385-b0e1-4ab5-916a-57992046f196\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:32.368923 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.368817 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b6b30385-b0e1-4ab5-916a-57992046f196-host-run-ovn-kubernetes\") pod \"ovnkube-node-pgnnf\" (UID: \"b6b30385-b0e1-4ab5-916a-57992046f196\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:32.368923 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.368845 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1f86594f-112f-4e58-a2ca-bbd951c4d1af-kubelet-config\") pod \"global-pull-secret-syncer-q2dm2\" (UID: \"1f86594f-112f-4e58-a2ca-bbd951c4d1af\") " pod="kube-system/global-pull-secret-syncer-q2dm2" Sep 11 20:00:32.368923 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.368867 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p6n2\" (UniqueName: \"kubernetes.io/projected/2133a2c0-f263-45aa-9d96-1819721a8c9b-kube-api-access-4p6n2\") pod \"node-resolver-2m6cs\" (UID: \"2133a2c0-f263-45aa-9d96-1819721a8c9b\") " pod="openshift-dns/node-resolver-2m6cs" Sep 11 20:00:32.369298 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.369004 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b6b30385-b0e1-4ab5-916a-57992046f196-run-openvswitch\") pod \"ovnkube-node-pgnnf\" (UID: \"b6b30385-b0e1-4ab5-916a-57992046f196\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:32.369298 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.369172 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b6b30385-b0e1-4ab5-916a-57992046f196-run-ovn\") pod \"ovnkube-node-pgnnf\" (UID: \"b6b30385-b0e1-4ab5-916a-57992046f196\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:32.369298 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.369261 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b6b30385-b0e1-4ab5-916a-57992046f196-log-socket\") pod \"ovnkube-node-pgnnf\" (UID: \"b6b30385-b0e1-4ab5-916a-57992046f196\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:32.369856 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.369829 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/52601b9c-beef-4b1c-aaf3-34cf267a00fe-registration-dir\") pod \"aws-ebs-csi-driver-node-w6p5s\" (UID: \"52601b9c-beef-4b1c-aaf3-34cf267a00fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w6p5s" Sep 11 20:00:32.369949 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.369871 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/171c51ac-8f0e-426f-973d-5a596e73d7d1-host-run-k8s-cni-cncf-io\") pod \"multus-8kcqg\" (UID: \"171c51ac-8f0e-426f-973d-5a596e73d7d1\") " pod="openshift-multus/multus-8kcqg" Sep 11 20:00:32.369949 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.369933 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b6b30385-b0e1-4ab5-916a-57992046f196-host-slash\") pod \"ovnkube-node-pgnnf\" (UID: \"b6b30385-b0e1-4ab5-916a-57992046f196\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:32.370019 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.369953 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b6b30385-b0e1-4ab5-916a-57992046f196-node-log\") pod \"ovnkube-node-pgnnf\" (UID: \"b6b30385-b0e1-4ab5-916a-57992046f196\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:32.370019 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.370002 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b6b30385-b0e1-4ab5-916a-57992046f196-host-cni-bin\") pod \"ovnkube-node-pgnnf\" (UID: \"b6b30385-b0e1-4ab5-916a-57992046f196\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:32.370110 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.370065 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1f86594f-112f-4e58-a2ca-bbd951c4d1af-original-pull-secret\") pod \"global-pull-secret-syncer-q2dm2\" (UID: \"1f86594f-112f-4e58-a2ca-bbd951c4d1af\") " pod="kube-system/global-pull-secret-syncer-q2dm2" Sep 11 20:00:32.370161 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.370107 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/52601b9c-beef-4b1c-aaf3-34cf267a00fe-device-dir\") pod \"aws-ebs-csi-driver-node-w6p5s\" (UID: \"52601b9c-beef-4b1c-aaf3-34cf267a00fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w6p5s" Sep 11 20:00:32.370161 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.370143 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/52601b9c-beef-4b1c-aaf3-34cf267a00fe-sys-fs\") pod \"aws-ebs-csi-driver-node-w6p5s\" (UID: \"52601b9c-beef-4b1c-aaf3-34cf267a00fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w6p5s" Sep 11 20:00:32.370254 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.370197 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/52c72e1a-d061-4d3c-922b-b572a6496fed-cnibin\") pod \"multus-additional-cni-plugins-ddmz9\" (UID: \"52c72e1a-d061-4d3c-922b-b572a6496fed\") " pod="openshift-multus/multus-additional-cni-plugins-ddmz9" Sep 11 20:00:32.370254 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.370229 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/52c72e1a-d061-4d3c-922b-b572a6496fed-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ddmz9\" (UID: \"52c72e1a-d061-4d3c-922b-b572a6496fed\") " pod="openshift-multus/multus-additional-cni-plugins-ddmz9" Sep 11 20:00:32.370347 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.370252 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-r64rv" Sep 11 20:00:32.370347 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.370275 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwhtf\" (UniqueName: \"kubernetes.io/projected/ae2b5d1f-4bb1-41eb-9096-d54b304ad672-kube-api-access-cwhtf\") pod \"network-check-target-wk9hk\" (UID: \"ae2b5d1f-4bb1-41eb-9096-d54b304ad672\") " pod="openshift-network-diagnostics/network-check-target-wk9hk" Sep 11 20:00:32.371153 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.370316 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b6b30385-b0e1-4ab5-916a-57992046f196-host-run-netns\") pod \"ovnkube-node-pgnnf\" (UID: \"b6b30385-b0e1-4ab5-916a-57992046f196\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:32.371153 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.370794 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2133a2c0-f263-45aa-9d96-1819721a8c9b-tmp-dir\") pod \"node-resolver-2m6cs\" (UID: \"2133a2c0-f263-45aa-9d96-1819721a8c9b\") " pod="openshift-dns/node-resolver-2m6cs" Sep 11 20:00:32.371153 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.370820 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/52601b9c-beef-4b1c-aaf3-34cf267a00fe-socket-dir\") pod \"aws-ebs-csi-driver-node-w6p5s\" (UID: \"52601b9c-beef-4b1c-aaf3-34cf267a00fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w6p5s" Sep 11 20:00:32.371153 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.370844 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/52601b9c-beef-4b1c-aaf3-34cf267a00fe-etc-selinux\") pod \"aws-ebs-csi-driver-node-w6p5s\" (UID: \"52601b9c-beef-4b1c-aaf3-34cf267a00fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w6p5s" Sep 11 20:00:32.371153 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.370866 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/171c51ac-8f0e-426f-973d-5a596e73d7d1-multus-socket-dir-parent\") pod \"multus-8kcqg\" (UID: \"171c51ac-8f0e-426f-973d-5a596e73d7d1\") " pod="openshift-multus/multus-8kcqg" Sep 11 20:00:32.371153 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.370888 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b6b30385-b0e1-4ab5-916a-57992046f196-run-systemd\") pod \"ovnkube-node-pgnnf\" (UID: \"b6b30385-b0e1-4ab5-916a-57992046f196\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:32.371153 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.370921 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b6b30385-b0e1-4ab5-916a-57992046f196-etc-openvswitch\") pod \"ovnkube-node-pgnnf\" (UID: \"b6b30385-b0e1-4ab5-916a-57992046f196\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:32.371153 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.370945 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pg6c\" (UniqueName: \"kubernetes.io/projected/52c72e1a-d061-4d3c-922b-b572a6496fed-kube-api-access-4pg6c\") pod \"multus-additional-cni-plugins-ddmz9\" (UID: \"52c72e1a-d061-4d3c-922b-b572a6496fed\") " pod="openshift-multus/multus-additional-cni-plugins-ddmz9" Sep 11 20:00:32.371153 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.370969 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b6b30385-b0e1-4ab5-916a-57992046f196-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pgnnf\" (UID: \"b6b30385-b0e1-4ab5-916a-57992046f196\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:32.371153 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.371000 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/52c72e1a-d061-4d3c-922b-b572a6496fed-cni-binary-copy\") pod \"multus-additional-cni-plugins-ddmz9\" (UID: \"52c72e1a-d061-4d3c-922b-b572a6496fed\") " pod="openshift-multus/multus-additional-cni-plugins-ddmz9" Sep 11 20:00:32.371153 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.371022 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/52c72e1a-d061-4d3c-922b-b572a6496fed-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ddmz9\" (UID: \"52c72e1a-d061-4d3c-922b-b572a6496fed\") " pod="openshift-multus/multus-additional-cni-plugins-ddmz9" Sep 11 20:00:32.371153 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.371055 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/52c72e1a-d061-4d3c-922b-b572a6496fed-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ddmz9\" (UID: \"52c72e1a-d061-4d3c-922b-b572a6496fed\") " pod="openshift-multus/multus-additional-cni-plugins-ddmz9" Sep 11 20:00:32.371153 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.371103 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b6b30385-b0e1-4ab5-916a-57992046f196-ovnkube-script-lib\") pod \"ovnkube-node-pgnnf\" (UID: \"b6b30385-b0e1-4ab5-916a-57992046f196\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:32.372982 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.372941 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 11 20:00:32.372982 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.372972 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 11 20:00:32.373111 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.373040 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tmjvz\"" Sep 11 20:00:32.373111 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.372942 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 11 20:00:32.373204 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.373122 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-f6skx" Sep 11 20:00:32.374979 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.374958 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 11 20:00:32.375214 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.375196 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7zncg\"" Sep 11 20:00:32.375327 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.375206 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 11 20:00:32.394524 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.394503 2546 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-mv48z" Sep 11 20:00:32.400682 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.400664 2546 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-mv48z" Sep 11 20:00:32.447708 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.447660 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-205.ec2.internal" event={"ID":"d907173ce62f903e18094a4e7845248d","Type":"ContainerStarted","Data":"2d1daa9f8f25a8cd807dac33771ed42f2f8b353ad6b496d335f9ce5c15ca3f55"} Sep 11 20:00:32.448829 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.448807 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-205.ec2.internal" event={"ID":"11ecaef2c363c5ae25ba196bd5dc1142","Type":"ContainerStarted","Data":"fbae09b8a86fa2ba561ba9fc744d4359825a104e4c5be3611180c22e212d12d1"} Sep 11 20:00:32.457379 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.457360 2546 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 11 20:00:32.466601 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.466566 2546 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 20:00:32.472252 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.472231 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/171c51ac-8f0e-426f-973d-5a596e73d7d1-host-run-k8s-cni-cncf-io\") pod \"multus-8kcqg\" (UID: \"171c51ac-8f0e-426f-973d-5a596e73d7d1\") " pod="openshift-multus/multus-8kcqg" Sep 11 20:00:32.472352 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.472260 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b6b30385-b0e1-4ab5-916a-57992046f196-host-slash\") pod \"ovnkube-node-pgnnf\" (UID: \"b6b30385-b0e1-4ab5-916a-57992046f196\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:32.472352 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.472283 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b6b30385-b0e1-4ab5-916a-57992046f196-node-log\") pod \"ovnkube-node-pgnnf\" (UID: \"b6b30385-b0e1-4ab5-916a-57992046f196\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:32.472352 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.472306 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b6b30385-b0e1-4ab5-916a-57992046f196-host-cni-bin\") pod \"ovnkube-node-pgnnf\" (UID: \"b6b30385-b0e1-4ab5-916a-57992046f196\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:32.472491 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.472364 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b6b30385-b0e1-4ab5-916a-57992046f196-host-cni-bin\") pod \"ovnkube-node-pgnnf\" (UID: \"b6b30385-b0e1-4ab5-916a-57992046f196\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:32.472491 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.472367 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/171c51ac-8f0e-426f-973d-5a596e73d7d1-host-run-k8s-cni-cncf-io\") pod \"multus-8kcqg\" (UID: \"171c51ac-8f0e-426f-973d-5a596e73d7d1\") " pod="openshift-multus/multus-8kcqg" Sep 11 20:00:32.472491 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.472366 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b6b30385-b0e1-4ab5-916a-57992046f196-host-slash\") pod \"ovnkube-node-pgnnf\" (UID: \"b6b30385-b0e1-4ab5-916a-57992046f196\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:32.472491 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.472389 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1f86594f-112f-4e58-a2ca-bbd951c4d1af-original-pull-secret\") pod \"global-pull-secret-syncer-q2dm2\" (UID: \"1f86594f-112f-4e58-a2ca-bbd951c4d1af\") " pod="kube-system/global-pull-secret-syncer-q2dm2" Sep 11 20:00:32.472491 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.472436 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/50ff3d16-16c7-4631-bf85-6ee04ed3b508-etc-modprobe-d\") pod \"tuned-f6skx\" (UID: \"50ff3d16-16c7-4631-bf85-6ee04ed3b508\") " pod="openshift-cluster-node-tuning-operator/tuned-f6skx" Sep 11 20:00:32.472491 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.472373 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b6b30385-b0e1-4ab5-916a-57992046f196-node-log\") pod \"ovnkube-node-pgnnf\" (UID: \"b6b30385-b0e1-4ab5-916a-57992046f196\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:32.472491 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.472461 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/50ff3d16-16c7-4631-bf85-6ee04ed3b508-host\") pod \"tuned-f6skx\" (UID: \"50ff3d16-16c7-4631-bf85-6ee04ed3b508\") " pod="openshift-cluster-node-tuning-operator/tuned-f6skx" Sep 11 20:00:32.472491 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:32.472479 2546 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 20:00:32.472491 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.472490 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/52601b9c-beef-4b1c-aaf3-34cf267a00fe-device-dir\") pod \"aws-ebs-csi-driver-node-w6p5s\" (UID: \"52601b9c-beef-4b1c-aaf3-34cf267a00fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w6p5s" Sep 11 20:00:32.472909 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.472513 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/52601b9c-beef-4b1c-aaf3-34cf267a00fe-sys-fs\") pod \"aws-ebs-csi-driver-node-w6p5s\" (UID: \"52601b9c-beef-4b1c-aaf3-34cf267a00fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w6p5s" Sep 11 20:00:32.472909 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:32.472557 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f86594f-112f-4e58-a2ca-bbd951c4d1af-original-pull-secret podName:1f86594f-112f-4e58-a2ca-bbd951c4d1af nodeName:}" failed. No retries permitted until 2026-09-11 20:00:32.972525642 +0000 UTC m=+3.020575975 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1f86594f-112f-4e58-a2ca-bbd951c4d1af-original-pull-secret") pod "global-pull-secret-syncer-q2dm2" (UID: "1f86594f-112f-4e58-a2ca-bbd951c4d1af") : object "kube-system"/"original-pull-secret" not registered Sep 11 20:00:32.472909 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.472595 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/52c72e1a-d061-4d3c-922b-b572a6496fed-cnibin\") pod \"multus-additional-cni-plugins-ddmz9\" (UID: \"52c72e1a-d061-4d3c-922b-b572a6496fed\") " pod="openshift-multus/multus-additional-cni-plugins-ddmz9" Sep 11 20:00:32.472909 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.472593 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/52601b9c-beef-4b1c-aaf3-34cf267a00fe-device-dir\") pod \"aws-ebs-csi-driver-node-w6p5s\" (UID: \"52601b9c-beef-4b1c-aaf3-34cf267a00fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w6p5s" Sep 11 20:00:32.472909 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.472604 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/52601b9c-beef-4b1c-aaf3-34cf267a00fe-sys-fs\") pod \"aws-ebs-csi-driver-node-w6p5s\" (UID: \"52601b9c-beef-4b1c-aaf3-34cf267a00fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w6p5s" Sep 11 20:00:32.472909 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.472625 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/52c72e1a-d061-4d3c-922b-b572a6496fed-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ddmz9\" (UID: \"52c72e1a-d061-4d3c-922b-b572a6496fed\") " pod="openshift-multus/multus-additional-cni-plugins-ddmz9" Sep 11 20:00:32.472909 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.472660 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/52c72e1a-d061-4d3c-922b-b572a6496fed-cnibin\") pod \"multus-additional-cni-plugins-ddmz9\" (UID: \"52c72e1a-d061-4d3c-922b-b572a6496fed\") " pod="openshift-multus/multus-additional-cni-plugins-ddmz9" Sep 11 20:00:32.472909 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.472667 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwhtf\" (UniqueName: \"kubernetes.io/projected/ae2b5d1f-4bb1-41eb-9096-d54b304ad672-kube-api-access-cwhtf\") pod \"network-check-target-wk9hk\" (UID: \"ae2b5d1f-4bb1-41eb-9096-d54b304ad672\") " pod="openshift-network-diagnostics/network-check-target-wk9hk" Sep 11 20:00:32.472909 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.472704 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6e8ee8b7-3b67-4b9d-88fc-552fd655d11a-serviceca\") pod \"node-ca-54fff\" (UID: \"6e8ee8b7-3b67-4b9d-88fc-552fd655d11a\") " pod="openshift-image-registry/node-ca-54fff" Sep 11 20:00:32.472909 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.472744 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b6b30385-b0e1-4ab5-916a-57992046f196-host-run-netns\") pod \"ovnkube-node-pgnnf\" (UID: \"b6b30385-b0e1-4ab5-916a-57992046f196\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:32.472909 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.472818 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b6b30385-b0e1-4ab5-916a-57992046f196-host-run-netns\") pod \"ovnkube-node-pgnnf\" (UID: \"b6b30385-b0e1-4ab5-916a-57992046f196\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:32.472909 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.472871 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2133a2c0-f263-45aa-9d96-1819721a8c9b-tmp-dir\") pod \"node-resolver-2m6cs\" (UID: \"2133a2c0-f263-45aa-9d96-1819721a8c9b\") " pod="openshift-dns/node-resolver-2m6cs" Sep 11 20:00:32.473486 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.472920 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/50ff3d16-16c7-4631-bf85-6ee04ed3b508-etc-sysconfig\") pod \"tuned-f6skx\" (UID: \"50ff3d16-16c7-4631-bf85-6ee04ed3b508\") " pod="openshift-cluster-node-tuning-operator/tuned-f6skx" Sep 11 20:00:32.473486 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.472964 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/52601b9c-beef-4b1c-aaf3-34cf267a00fe-socket-dir\") pod \"aws-ebs-csi-driver-node-w6p5s\" (UID: \"52601b9c-beef-4b1c-aaf3-34cf267a00fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w6p5s" Sep 11 20:00:32.473486 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.472989 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/52601b9c-beef-4b1c-aaf3-34cf267a00fe-etc-selinux\") pod \"aws-ebs-csi-driver-node-w6p5s\" (UID: \"52601b9c-beef-4b1c-aaf3-34cf267a00fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w6p5s" Sep 11 20:00:32.473486 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.473015 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/171c51ac-8f0e-426f-973d-5a596e73d7d1-multus-socket-dir-parent\") pod \"multus-8kcqg\" (UID: \"171c51ac-8f0e-426f-973d-5a596e73d7d1\") " pod="openshift-multus/multus-8kcqg" Sep 11 20:00:32.473486 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.473038 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b6b30385-b0e1-4ab5-916a-57992046f196-run-systemd\") pod \"ovnkube-node-pgnnf\" (UID: \"b6b30385-b0e1-4ab5-916a-57992046f196\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:32.473486 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.473061 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b6b30385-b0e1-4ab5-916a-57992046f196-etc-openvswitch\") pod \"ovnkube-node-pgnnf\" (UID: \"b6b30385-b0e1-4ab5-916a-57992046f196\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:32.473486 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.473101 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pg6c\" (UniqueName: \"kubernetes.io/projected/52c72e1a-d061-4d3c-922b-b572a6496fed-kube-api-access-4pg6c\") pod \"multus-additional-cni-plugins-ddmz9\" (UID: \"52c72e1a-d061-4d3c-922b-b572a6496fed\") " pod="openshift-multus/multus-additional-cni-plugins-ddmz9" Sep 11 20:00:32.473486 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.473112 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/171c51ac-8f0e-426f-973d-5a596e73d7d1-multus-socket-dir-parent\") pod \"multus-8kcqg\" (UID: \"171c51ac-8f0e-426f-973d-5a596e73d7d1\") " pod="openshift-multus/multus-8kcqg" Sep 11 20:00:32.473486 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.473126 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce96d552-27b5-4912-86b0-38f0f46d0002-metrics-certs\") pod \"network-metrics-daemon-zpl5q\" (UID: \"ce96d552-27b5-4912-86b0-38f0f46d0002\") " pod="openshift-multus/network-metrics-daemon-zpl5q" Sep 11 20:00:32.473486 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.473129 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2133a2c0-f263-45aa-9d96-1819721a8c9b-tmp-dir\") pod \"node-resolver-2m6cs\" (UID: \"2133a2c0-f263-45aa-9d96-1819721a8c9b\") " pod="openshift-dns/node-resolver-2m6cs" Sep 11 20:00:32.473486 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.473142 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/52601b9c-beef-4b1c-aaf3-34cf267a00fe-socket-dir\") pod \"aws-ebs-csi-driver-node-w6p5s\" (UID: \"52601b9c-beef-4b1c-aaf3-34cf267a00fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w6p5s" Sep 11 20:00:32.473486 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.473156 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b6b30385-b0e1-4ab5-916a-57992046f196-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pgnnf\" (UID: \"b6b30385-b0e1-4ab5-916a-57992046f196\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:32.473486 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.473186 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/52c72e1a-d061-4d3c-922b-b572a6496fed-cni-binary-copy\") pod \"multus-additional-cni-plugins-ddmz9\" (UID: \"52c72e1a-d061-4d3c-922b-b572a6496fed\") " pod="openshift-multus/multus-additional-cni-plugins-ddmz9" Sep 11 20:00:32.473486 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.473205 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b6b30385-b0e1-4ab5-916a-57992046f196-etc-openvswitch\") pod \"ovnkube-node-pgnnf\" (UID: \"b6b30385-b0e1-4ab5-916a-57992046f196\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:32.473486 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.473205 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b6b30385-b0e1-4ab5-916a-57992046f196-run-systemd\") pod \"ovnkube-node-pgnnf\" (UID: \"b6b30385-b0e1-4ab5-916a-57992046f196\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:32.473486 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.473231 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/52c72e1a-d061-4d3c-922b-b572a6496fed-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ddmz9\" (UID: \"52c72e1a-d061-4d3c-922b-b572a6496fed\") " pod="openshift-multus/multus-additional-cni-plugins-ddmz9" Sep 11 20:00:32.473486 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.473223 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/52601b9c-beef-4b1c-aaf3-34cf267a00fe-etc-selinux\") pod \"aws-ebs-csi-driver-node-w6p5s\" (UID: \"52601b9c-beef-4b1c-aaf3-34cf267a00fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w6p5s" Sep 11 20:00:32.474351 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.473257 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/52c72e1a-d061-4d3c-922b-b572a6496fed-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ddmz9\" (UID: \"52c72e1a-d061-4d3c-922b-b572a6496fed\") " pod="openshift-multus/multus-additional-cni-plugins-ddmz9" Sep 11 20:00:32.474351 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.473263 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b6b30385-b0e1-4ab5-916a-57992046f196-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pgnnf\" (UID: \"b6b30385-b0e1-4ab5-916a-57992046f196\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:32.474351 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.473293 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/52c72e1a-d061-4d3c-922b-b572a6496fed-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ddmz9\" (UID: \"52c72e1a-d061-4d3c-922b-b572a6496fed\") " pod="openshift-multus/multus-additional-cni-plugins-ddmz9" Sep 11 20:00:32.474351 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.473280 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6e8ee8b7-3b67-4b9d-88fc-552fd655d11a-host\") pod \"node-ca-54fff\" (UID: \"6e8ee8b7-3b67-4b9d-88fc-552fd655d11a\") " pod="openshift-image-registry/node-ca-54fff" Sep 11 20:00:32.474351 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.473365 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z872\" (UniqueName: \"kubernetes.io/projected/6e8ee8b7-3b67-4b9d-88fc-552fd655d11a-kube-api-access-2z872\") pod \"node-ca-54fff\" (UID: \"6e8ee8b7-3b67-4b9d-88fc-552fd655d11a\") " pod="openshift-image-registry/node-ca-54fff" Sep 11 20:00:32.474351 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.473389 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b6b30385-b0e1-4ab5-916a-57992046f196-ovnkube-script-lib\") pod \"ovnkube-node-pgnnf\" (UID: \"b6b30385-b0e1-4ab5-916a-57992046f196\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:32.474351 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.473390 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/52c72e1a-d061-4d3c-922b-b572a6496fed-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ddmz9\" (UID: \"52c72e1a-d061-4d3c-922b-b572a6496fed\") " pod="openshift-multus/multus-additional-cni-plugins-ddmz9" Sep 11 20:00:32.474351 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.473421 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2pgj\" (UniqueName: \"kubernetes.io/projected/52601b9c-beef-4b1c-aaf3-34cf267a00fe-kube-api-access-h2pgj\") pod \"aws-ebs-csi-driver-node-w6p5s\" (UID: \"52601b9c-beef-4b1c-aaf3-34cf267a00fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w6p5s" Sep 11 20:00:32.474351 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.473449 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/171c51ac-8f0e-426f-973d-5a596e73d7d1-host-var-lib-cni-bin\") pod \"multus-8kcqg\" (UID: \"171c51ac-8f0e-426f-973d-5a596e73d7d1\") " pod="openshift-multus/multus-8kcqg" Sep 11 20:00:32.474351 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.473474 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88sjc\" (UniqueName: \"kubernetes.io/projected/171c51ac-8f0e-426f-973d-5a596e73d7d1-kube-api-access-88sjc\") pod \"multus-8kcqg\" (UID: \"171c51ac-8f0e-426f-973d-5a596e73d7d1\") " pod="openshift-multus/multus-8kcqg" Sep 11 20:00:32.474351 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.473500 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2133a2c0-f263-45aa-9d96-1819721a8c9b-hosts-file\") pod \"node-resolver-2m6cs\" (UID: \"2133a2c0-f263-45aa-9d96-1819721a8c9b\") " pod="openshift-dns/node-resolver-2m6cs" Sep 11 20:00:32.474351 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.473524 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/171c51ac-8f0e-426f-973d-5a596e73d7d1-system-cni-dir\") pod \"multus-8kcqg\" (UID: \"171c51ac-8f0e-426f-973d-5a596e73d7d1\") " pod="openshift-multus/multus-8kcqg" Sep 11 20:00:32.474351 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.473548 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/171c51ac-8f0e-426f-973d-5a596e73d7d1-multus-cni-dir\") pod \"multus-8kcqg\" (UID: \"171c51ac-8f0e-426f-973d-5a596e73d7d1\") " pod="openshift-multus/multus-8kcqg" Sep 11 20:00:32.474351 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.473570 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/171c51ac-8f0e-426f-973d-5a596e73d7d1-host-var-lib-kubelet\") pod \"multus-8kcqg\" (UID: \"171c51ac-8f0e-426f-973d-5a596e73d7d1\") " pod="openshift-multus/multus-8kcqg" Sep 11 20:00:32.474351 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.473609 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/171c51ac-8f0e-426f-973d-5a596e73d7d1-multus-daemon-config\") pod \"multus-8kcqg\" (UID: \"171c51ac-8f0e-426f-973d-5a596e73d7d1\") " pod="openshift-multus/multus-8kcqg" Sep 11 20:00:32.474351 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.473632 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b6b30385-b0e1-4ab5-916a-57992046f196-ovn-node-metrics-cert\") pod \"ovnkube-node-pgnnf\" (UID: \"b6b30385-b0e1-4ab5-916a-57992046f196\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:32.474351 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.473657 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/50ff3d16-16c7-4631-bf85-6ee04ed3b508-etc-systemd\") pod \"tuned-f6skx\" (UID: \"50ff3d16-16c7-4631-bf85-6ee04ed3b508\") " pod="openshift-cluster-node-tuning-operator/tuned-f6skx" Sep 11 20:00:32.475163 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.473683 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50ff3d16-16c7-4631-bf85-6ee04ed3b508-sys\") pod \"tuned-f6skx\" (UID: \"50ff3d16-16c7-4631-bf85-6ee04ed3b508\") " pod="openshift-cluster-node-tuning-operator/tuned-f6skx" Sep 11 20:00:32.475163 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.473709 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b6b30385-b0e1-4ab5-916a-57992046f196-host-cni-netd\") pod \"ovnkube-node-pgnnf\" (UID: \"b6b30385-b0e1-4ab5-916a-57992046f196\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:32.475163 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.473726 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/171c51ac-8f0e-426f-973d-5a596e73d7d1-multus-cni-dir\") pod \"multus-8kcqg\" (UID: \"171c51ac-8f0e-426f-973d-5a596e73d7d1\") " pod="openshift-multus/multus-8kcqg" Sep 11 20:00:32.475163 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.473731 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/50ff3d16-16c7-4631-bf85-6ee04ed3b508-lib-modules\") pod \"tuned-f6skx\" (UID: \"50ff3d16-16c7-4631-bf85-6ee04ed3b508\") " pod="openshift-cluster-node-tuning-operator/tuned-f6skx" Sep 11 20:00:32.475163 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.473769 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/52c72e1a-d061-4d3c-922b-b572a6496fed-cni-binary-copy\") pod \"multus-additional-cni-plugins-ddmz9\" (UID: \"52c72e1a-d061-4d3c-922b-b572a6496fed\") " pod="openshift-multus/multus-additional-cni-plugins-ddmz9" Sep 11 20:00:32.475163 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.473787 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/171c51ac-8f0e-426f-973d-5a596e73d7d1-host-var-lib-kubelet\") pod \"multus-8kcqg\" (UID: \"171c51ac-8f0e-426f-973d-5a596e73d7d1\") " pod="openshift-multus/multus-8kcqg" Sep 11 20:00:32.475163 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.473818 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/171c51ac-8f0e-426f-973d-5a596e73d7d1-host-var-lib-cni-bin\") pod \"multus-8kcqg\" (UID: \"171c51ac-8f0e-426f-973d-5a596e73d7d1\") " pod="openshift-multus/multus-8kcqg" Sep 11 20:00:32.475163 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.473830 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/171c51ac-8f0e-426f-973d-5a596e73d7d1-system-cni-dir\") pod \"multus-8kcqg\" (UID: \"171c51ac-8f0e-426f-973d-5a596e73d7d1\") " pod="openshift-multus/multus-8kcqg" Sep 11 20:00:32.475163 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.473837 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2133a2c0-f263-45aa-9d96-1819721a8c9b-hosts-file\") pod \"node-resolver-2m6cs\" (UID: \"2133a2c0-f263-45aa-9d96-1819721a8c9b\") " pod="openshift-dns/node-resolver-2m6cs" Sep 11 20:00:32.475163 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.473849 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b6b30385-b0e1-4ab5-916a-57992046f196-ovnkube-script-lib\") pod \"ovnkube-node-pgnnf\" (UID: \"b6b30385-b0e1-4ab5-916a-57992046f196\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:32.475163 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.473960 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b6b30385-b0e1-4ab5-916a-57992046f196-host-cni-netd\") pod \"ovnkube-node-pgnnf\" (UID: \"b6b30385-b0e1-4ab5-916a-57992046f196\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:32.475163 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.474001 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b54lr\" (UniqueName: \"kubernetes.io/projected/ce96d552-27b5-4912-86b0-38f0f46d0002-kube-api-access-b54lr\") pod \"network-metrics-daemon-zpl5q\" (UID: \"ce96d552-27b5-4912-86b0-38f0f46d0002\") " pod="openshift-multus/network-metrics-daemon-zpl5q" Sep 11 20:00:32.475163 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.474029 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/482b4b76-dc42-490f-afcb-5b672806c329-iptables-alerter-script\") pod \"iptables-alerter-r64rv\" (UID: \"482b4b76-dc42-490f-afcb-5b672806c329\") " pod="openshift-network-operator/iptables-alerter-r64rv" Sep 11 20:00:32.475163 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.474052 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc2r6\" (UniqueName: \"kubernetes.io/projected/482b4b76-dc42-490f-afcb-5b672806c329-kube-api-access-mc2r6\") pod \"iptables-alerter-r64rv\" (UID: \"482b4b76-dc42-490f-afcb-5b672806c329\") " pod="openshift-network-operator/iptables-alerter-r64rv" Sep 11 20:00:32.475163 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.474079 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/171c51ac-8f0e-426f-973d-5a596e73d7d1-host-var-lib-cni-multus\") pod \"multus-8kcqg\" (UID: \"171c51ac-8f0e-426f-973d-5a596e73d7d1\") " pod="openshift-multus/multus-8kcqg" Sep 11 20:00:32.475163 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.474106 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/171c51ac-8f0e-426f-973d-5a596e73d7d1-hostroot\") pod \"multus-8kcqg\" (UID: \"171c51ac-8f0e-426f-973d-5a596e73d7d1\") " pod="openshift-multus/multus-8kcqg" Sep 11 20:00:32.475163 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.474130 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd9x6\" (UniqueName: \"kubernetes.io/projected/b6b30385-b0e1-4ab5-916a-57992046f196-kube-api-access-gd9x6\") pod \"ovnkube-node-pgnnf\" (UID: \"b6b30385-b0e1-4ab5-916a-57992046f196\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:32.475951 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.474153 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1ab887bb-e9fb-4ad3-9178-a35ed6fecfed-konnectivity-ca\") pod \"konnectivity-agent-jdhk4\" (UID: \"1ab887bb-e9fb-4ad3-9178-a35ed6fecfed\") " pod="kube-system/konnectivity-agent-jdhk4" Sep 11 20:00:32.475951 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.474175 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/50ff3d16-16c7-4631-bf85-6ee04ed3b508-etc-sysctl-conf\") pod \"tuned-f6skx\" (UID: \"50ff3d16-16c7-4631-bf85-6ee04ed3b508\") " pod="openshift-cluster-node-tuning-operator/tuned-f6skx" Sep 11 20:00:32.475951 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.474197 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/50ff3d16-16c7-4631-bf85-6ee04ed3b508-run\") pod \"tuned-f6skx\" (UID: \"50ff3d16-16c7-4631-bf85-6ee04ed3b508\") " pod="openshift-cluster-node-tuning-operator/tuned-f6skx" Sep 11 20:00:32.475951 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.474220 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/52601b9c-beef-4b1c-aaf3-34cf267a00fe-kubelet-dir\") pod \"aws-ebs-csi-driver-node-w6p5s\" (UID: \"52601b9c-beef-4b1c-aaf3-34cf267a00fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w6p5s" Sep 11 20:00:32.475951 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.474250 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/171c51ac-8f0e-426f-973d-5a596e73d7d1-cnibin\") pod \"multus-8kcqg\" (UID: \"171c51ac-8f0e-426f-973d-5a596e73d7d1\") " pod="openshift-multus/multus-8kcqg" Sep 11 20:00:32.475951 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.474274 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/171c51ac-8f0e-426f-973d-5a596e73d7d1-host-run-multus-certs\") pod \"multus-8kcqg\" (UID: \"171c51ac-8f0e-426f-973d-5a596e73d7d1\") " pod="openshift-multus/multus-8kcqg" Sep 11 20:00:32.475951 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.474296 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/482b4b76-dc42-490f-afcb-5b672806c329-host-slash\") pod \"iptables-alerter-r64rv\" (UID: \"482b4b76-dc42-490f-afcb-5b672806c329\") " pod="openshift-network-operator/iptables-alerter-r64rv" Sep 11 20:00:32.475951 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.474330 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/171c51ac-8f0e-426f-973d-5a596e73d7d1-cni-binary-copy\") pod \"multus-8kcqg\" (UID: \"171c51ac-8f0e-426f-973d-5a596e73d7d1\") " pod="openshift-multus/multus-8kcqg" Sep 11 20:00:32.475951 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.474345 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/171c51ac-8f0e-426f-973d-5a596e73d7d1-multus-daemon-config\") pod \"multus-8kcqg\" (UID: \"171c51ac-8f0e-426f-973d-5a596e73d7d1\") " pod="openshift-multus/multus-8kcqg" Sep 11 20:00:32.475951 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.474354 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/171c51ac-8f0e-426f-973d-5a596e73d7d1-multus-conf-dir\") pod \"multus-8kcqg\" (UID: \"171c51ac-8f0e-426f-973d-5a596e73d7d1\") " pod="openshift-multus/multus-8kcqg" Sep 11 20:00:32.475951 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.474362 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/52c72e1a-d061-4d3c-922b-b572a6496fed-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ddmz9\" (UID: \"52c72e1a-d061-4d3c-922b-b572a6496fed\") " pod="openshift-multus/multus-additional-cni-plugins-ddmz9" Sep 11 20:00:32.475951 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.474379 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b6b30385-b0e1-4ab5-916a-57992046f196-systemd-units\") pod \"ovnkube-node-pgnnf\" (UID: \"b6b30385-b0e1-4ab5-916a-57992046f196\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:32.475951 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.474423 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b6b30385-b0e1-4ab5-916a-57992046f196-systemd-units\") pod \"ovnkube-node-pgnnf\" (UID: \"b6b30385-b0e1-4ab5-916a-57992046f196\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:32.475951 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.474457 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b6b30385-b0e1-4ab5-916a-57992046f196-var-lib-openvswitch\") pod \"ovnkube-node-pgnnf\" (UID: \"b6b30385-b0e1-4ab5-916a-57992046f196\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:32.475951 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.474462 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/171c51ac-8f0e-426f-973d-5a596e73d7d1-host-var-lib-cni-multus\") pod \"multus-8kcqg\" (UID: \"171c51ac-8f0e-426f-973d-5a596e73d7d1\") " pod="openshift-multus/multus-8kcqg" Sep 11 20:00:32.475951 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.474482 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b6b30385-b0e1-4ab5-916a-57992046f196-ovnkube-config\") pod \"ovnkube-node-pgnnf\" (UID: \"b6b30385-b0e1-4ab5-916a-57992046f196\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:32.475951 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.474507 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1ab887bb-e9fb-4ad3-9178-a35ed6fecfed-agent-certs\") pod \"konnectivity-agent-jdhk4\" (UID: \"1ab887bb-e9fb-4ad3-9178-a35ed6fecfed\") " pod="kube-system/konnectivity-agent-jdhk4" Sep 11 20:00:32.476769 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.474511 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/171c51ac-8f0e-426f-973d-5a596e73d7d1-host-run-multus-certs\") pod \"multus-8kcqg\" (UID: \"171c51ac-8f0e-426f-973d-5a596e73d7d1\") " pod="openshift-multus/multus-8kcqg" Sep 11 20:00:32.476769 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.474537 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/50ff3d16-16c7-4631-bf85-6ee04ed3b508-etc-sysctl-d\") pod \"tuned-f6skx\" (UID: \"50ff3d16-16c7-4631-bf85-6ee04ed3b508\") " pod="openshift-cluster-node-tuning-operator/tuned-f6skx" Sep 11 20:00:32.476769 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.474558 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/52601b9c-beef-4b1c-aaf3-34cf267a00fe-kubelet-dir\") pod \"aws-ebs-csi-driver-node-w6p5s\" (UID: \"52601b9c-beef-4b1c-aaf3-34cf267a00fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w6p5s" Sep 11 20:00:32.476769 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.474562 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/50ff3d16-16c7-4631-bf85-6ee04ed3b508-etc-tuned\") pod \"tuned-f6skx\" (UID: \"50ff3d16-16c7-4631-bf85-6ee04ed3b508\") " pod="openshift-cluster-node-tuning-operator/tuned-f6skx" Sep 11 20:00:32.476769 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.474458 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/171c51ac-8f0e-426f-973d-5a596e73d7d1-hostroot\") pod \"multus-8kcqg\" (UID: \"171c51ac-8f0e-426f-973d-5a596e73d7d1\") " pod="openshift-multus/multus-8kcqg" Sep 11 20:00:32.476769 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.474620 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/171c51ac-8f0e-426f-973d-5a596e73d7d1-host-run-netns\") pod \"multus-8kcqg\" (UID: \"171c51ac-8f0e-426f-973d-5a596e73d7d1\") " pod="openshift-multus/multus-8kcqg" Sep 11 20:00:32.476769 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.474630 2546 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 11 20:00:32.476769 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.474648 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/171c51ac-8f0e-426f-973d-5a596e73d7d1-etc-kubernetes\") pod \"multus-8kcqg\" (UID: \"171c51ac-8f0e-426f-973d-5a596e73d7d1\") " pod="openshift-multus/multus-8kcqg" Sep 11 20:00:32.476769 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.474818 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b6b30385-b0e1-4ab5-916a-57992046f196-env-overrides\") pod \"ovnkube-node-pgnnf\" (UID: \"b6b30385-b0e1-4ab5-916a-57992046f196\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:32.476769 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.474996 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/171c51ac-8f0e-426f-973d-5a596e73d7d1-cnibin\") pod \"multus-8kcqg\" (UID: \"171c51ac-8f0e-426f-973d-5a596e73d7d1\") " pod="openshift-multus/multus-8kcqg" Sep 11 20:00:32.476769 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.475025 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b6b30385-b0e1-4ab5-916a-57992046f196-var-lib-openvswitch\") pod \"ovnkube-node-pgnnf\" (UID: \"b6b30385-b0e1-4ab5-916a-57992046f196\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:32.476769 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.475372 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/171c51ac-8f0e-426f-973d-5a596e73d7d1-cni-binary-copy\") pod \"multus-8kcqg\" (UID: \"171c51ac-8f0e-426f-973d-5a596e73d7d1\") " pod="openshift-multus/multus-8kcqg" Sep 11 20:00:32.476769 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.475385 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b6b30385-b0e1-4ab5-916a-57992046f196-ovnkube-config\") pod \"ovnkube-node-pgnnf\" (UID: \"b6b30385-b0e1-4ab5-916a-57992046f196\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:32.476769 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.475415 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/171c51ac-8f0e-426f-973d-5a596e73d7d1-multus-conf-dir\") pod \"multus-8kcqg\" (UID: \"171c51ac-8f0e-426f-973d-5a596e73d7d1\") " pod="openshift-multus/multus-8kcqg" Sep 11 20:00:32.476769 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.475466 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/171c51ac-8f0e-426f-973d-5a596e73d7d1-etc-kubernetes\") pod \"multus-8kcqg\" (UID: \"171c51ac-8f0e-426f-973d-5a596e73d7d1\") " pod="openshift-multus/multus-8kcqg" Sep 11 20:00:32.476769 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.475496 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/171c51ac-8f0e-426f-973d-5a596e73d7d1-host-run-netns\") pod \"multus-8kcqg\" (UID: \"171c51ac-8f0e-426f-973d-5a596e73d7d1\") " pod="openshift-multus/multus-8kcqg" Sep 11 20:00:32.476769 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.475564 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1f86594f-112f-4e58-a2ca-bbd951c4d1af-dbus\") pod \"global-pull-secret-syncer-q2dm2\" (UID: \"1f86594f-112f-4e58-a2ca-bbd951c4d1af\") " pod="kube-system/global-pull-secret-syncer-q2dm2" Sep 11 20:00:32.476769 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.475667 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1f86594f-112f-4e58-a2ca-bbd951c4d1af-dbus\") pod \"global-pull-secret-syncer-q2dm2\" (UID: \"1f86594f-112f-4e58-a2ca-bbd951c4d1af\") " pod="kube-system/global-pull-secret-syncer-q2dm2" Sep 11 20:00:32.477534 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.475705 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/50ff3d16-16c7-4631-bf85-6ee04ed3b508-var-lib-kubelet\") pod \"tuned-f6skx\" (UID: \"50ff3d16-16c7-4631-bf85-6ee04ed3b508\") " pod="openshift-cluster-node-tuning-operator/tuned-f6skx" Sep 11 20:00:32.477534 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.475742 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/52c72e1a-d061-4d3c-922b-b572a6496fed-system-cni-dir\") pod \"multus-additional-cni-plugins-ddmz9\" (UID: \"52c72e1a-d061-4d3c-922b-b572a6496fed\") " pod="openshift-multus/multus-additional-cni-plugins-ddmz9" Sep 11 20:00:32.477534 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.475776 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/50ff3d16-16c7-4631-bf85-6ee04ed3b508-etc-kubernetes\") pod \"tuned-f6skx\" (UID: \"50ff3d16-16c7-4631-bf85-6ee04ed3b508\") " pod="openshift-cluster-node-tuning-operator/tuned-f6skx" Sep 11 20:00:32.477534 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.475807 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s645j\" (UniqueName: \"kubernetes.io/projected/50ff3d16-16c7-4631-bf85-6ee04ed3b508-kube-api-access-s645j\") pod \"tuned-f6skx\" (UID: \"50ff3d16-16c7-4631-bf85-6ee04ed3b508\") " pod="openshift-cluster-node-tuning-operator/tuned-f6skx" Sep 11 20:00:32.477534 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.475839 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6b30385-b0e1-4ab5-916a-57992046f196-host-kubelet\") pod \"ovnkube-node-pgnnf\" (UID: \"b6b30385-b0e1-4ab5-916a-57992046f196\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:32.477534 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.475852 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b6b30385-b0e1-4ab5-916a-57992046f196-env-overrides\") pod \"ovnkube-node-pgnnf\" (UID: \"b6b30385-b0e1-4ab5-916a-57992046f196\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:32.477534 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.475867 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b6b30385-b0e1-4ab5-916a-57992046f196-host-run-ovn-kubernetes\") pod \"ovnkube-node-pgnnf\" (UID: \"b6b30385-b0e1-4ab5-916a-57992046f196\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:32.477534 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.475901 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1f86594f-112f-4e58-a2ca-bbd951c4d1af-kubelet-config\") pod \"global-pull-secret-syncer-q2dm2\" (UID: \"1f86594f-112f-4e58-a2ca-bbd951c4d1af\") " pod="kube-system/global-pull-secret-syncer-q2dm2" Sep 11 20:00:32.477534 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.475986 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1f86594f-112f-4e58-a2ca-bbd951c4d1af-kubelet-config\") pod \"global-pull-secret-syncer-q2dm2\" (UID: \"1f86594f-112f-4e58-a2ca-bbd951c4d1af\") " pod="kube-system/global-pull-secret-syncer-q2dm2" Sep 11 20:00:32.477534 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.476025 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6b30385-b0e1-4ab5-916a-57992046f196-host-kubelet\") pod \"ovnkube-node-pgnnf\" (UID: \"b6b30385-b0e1-4ab5-916a-57992046f196\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:32.477534 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.476032 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b6b30385-b0e1-4ab5-916a-57992046f196-host-run-ovn-kubernetes\") pod \"ovnkube-node-pgnnf\" (UID: \"b6b30385-b0e1-4ab5-916a-57992046f196\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:32.477534 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.476053 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p6n2\" (UniqueName: \"kubernetes.io/projected/2133a2c0-f263-45aa-9d96-1819721a8c9b-kube-api-access-4p6n2\") pod \"node-resolver-2m6cs\" (UID: \"2133a2c0-f263-45aa-9d96-1819721a8c9b\") " pod="openshift-dns/node-resolver-2m6cs" Sep 11 20:00:32.477534 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.476117 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b6b30385-b0e1-4ab5-916a-57992046f196-run-openvswitch\") pod \"ovnkube-node-pgnnf\" (UID: \"b6b30385-b0e1-4ab5-916a-57992046f196\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:32.477534 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.476232 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b6b30385-b0e1-4ab5-916a-57992046f196-run-ovn\") pod \"ovnkube-node-pgnnf\" (UID: \"b6b30385-b0e1-4ab5-916a-57992046f196\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:32.477534 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.476265 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b6b30385-b0e1-4ab5-916a-57992046f196-log-socket\") pod \"ovnkube-node-pgnnf\" (UID: \"b6b30385-b0e1-4ab5-916a-57992046f196\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:32.477534 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.476318 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50ff3d16-16c7-4631-bf85-6ee04ed3b508-tmp\") pod \"tuned-f6skx\" (UID: \"50ff3d16-16c7-4631-bf85-6ee04ed3b508\") " pod="openshift-cluster-node-tuning-operator/tuned-f6skx" Sep 11 20:00:32.477534 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.476342 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b6b30385-b0e1-4ab5-916a-57992046f196-run-openvswitch\") pod \"ovnkube-node-pgnnf\" (UID: \"b6b30385-b0e1-4ab5-916a-57992046f196\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:32.478348 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.476348 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/52601b9c-beef-4b1c-aaf3-34cf267a00fe-registration-dir\") pod \"aws-ebs-csi-driver-node-w6p5s\" (UID: \"52601b9c-beef-4b1c-aaf3-34cf267a00fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w6p5s" Sep 11 20:00:32.478348 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.476395 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/52c72e1a-d061-4d3c-922b-b572a6496fed-system-cni-dir\") pod \"multus-additional-cni-plugins-ddmz9\" (UID: \"52c72e1a-d061-4d3c-922b-b572a6496fed\") " pod="openshift-multus/multus-additional-cni-plugins-ddmz9" Sep 11 20:00:32.478348 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.476453 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b6b30385-b0e1-4ab5-916a-57992046f196-log-socket\") pod \"ovnkube-node-pgnnf\" (UID: \"b6b30385-b0e1-4ab5-916a-57992046f196\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:32.478348 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.477470 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b6b30385-b0e1-4ab5-916a-57992046f196-run-ovn\") pod \"ovnkube-node-pgnnf\" (UID: \"b6b30385-b0e1-4ab5-916a-57992046f196\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:32.478348 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.477557 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/52601b9c-beef-4b1c-aaf3-34cf267a00fe-registration-dir\") pod \"aws-ebs-csi-driver-node-w6p5s\" (UID: \"52601b9c-beef-4b1c-aaf3-34cf267a00fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w6p5s" Sep 11 20:00:32.479909 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.479747 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b6b30385-b0e1-4ab5-916a-57992046f196-ovn-node-metrics-cert\") pod \"ovnkube-node-pgnnf\" (UID: \"b6b30385-b0e1-4ab5-916a-57992046f196\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:32.480626 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:32.480608 2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 20:00:32.480716 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:32.480634 2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 20:00:32.480716 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:32.480649 2546 projected.go:196] Error preparing data for projected volume kube-api-access-cwhtf for pod openshift-network-diagnostics/network-check-target-wk9hk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 20:00:32.480838 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:32.480725 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae2b5d1f-4bb1-41eb-9096-d54b304ad672-kube-api-access-cwhtf podName:ae2b5d1f-4bb1-41eb-9096-d54b304ad672 nodeName:}" failed. No retries permitted until 2026-09-11 20:00:32.980707381 +0000 UTC m=+3.028757720 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-cwhtf" (UniqueName: "kubernetes.io/projected/ae2b5d1f-4bb1-41eb-9096-d54b304ad672-kube-api-access-cwhtf") pod "network-check-target-wk9hk" (UID: "ae2b5d1f-4bb1-41eb-9096-d54b304ad672") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 20:00:32.482418 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.482390 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88sjc\" (UniqueName: \"kubernetes.io/projected/171c51ac-8f0e-426f-973d-5a596e73d7d1-kube-api-access-88sjc\") pod \"multus-8kcqg\" (UID: \"171c51ac-8f0e-426f-973d-5a596e73d7d1\") " pod="openshift-multus/multus-8kcqg" Sep 11 20:00:32.482662 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.482630 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd9x6\" (UniqueName: \"kubernetes.io/projected/b6b30385-b0e1-4ab5-916a-57992046f196-kube-api-access-gd9x6\") pod \"ovnkube-node-pgnnf\" (UID: \"b6b30385-b0e1-4ab5-916a-57992046f196\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:32.483053 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.483030 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pg6c\" (UniqueName: \"kubernetes.io/projected/52c72e1a-d061-4d3c-922b-b572a6496fed-kube-api-access-4pg6c\") pod \"multus-additional-cni-plugins-ddmz9\" (UID: \"52c72e1a-d061-4d3c-922b-b572a6496fed\") " pod="openshift-multus/multus-additional-cni-plugins-ddmz9" Sep 11 20:00:32.483155 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.483059 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2pgj\" (UniqueName: \"kubernetes.io/projected/52601b9c-beef-4b1c-aaf3-34cf267a00fe-kube-api-access-h2pgj\") pod \"aws-ebs-csi-driver-node-w6p5s\" (UID: \"52601b9c-beef-4b1c-aaf3-34cf267a00fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w6p5s" Sep 11 20:00:32.484522 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.484502 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p6n2\" (UniqueName: \"kubernetes.io/projected/2133a2c0-f263-45aa-9d96-1819721a8c9b-kube-api-access-4p6n2\") pod \"node-resolver-2m6cs\" (UID: \"2133a2c0-f263-45aa-9d96-1819721a8c9b\") " pod="openshift-dns/node-resolver-2m6cs" Sep 11 20:00:32.577633 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.577591 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/50ff3d16-16c7-4631-bf85-6ee04ed3b508-etc-modprobe-d\") pod \"tuned-f6skx\" (UID: \"50ff3d16-16c7-4631-bf85-6ee04ed3b508\") " pod="openshift-cluster-node-tuning-operator/tuned-f6skx" Sep 11 20:00:32.577633 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.577634 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/50ff3d16-16c7-4631-bf85-6ee04ed3b508-host\") pod \"tuned-f6skx\" (UID: \"50ff3d16-16c7-4631-bf85-6ee04ed3b508\") " pod="openshift-cluster-node-tuning-operator/tuned-f6skx" Sep 11 20:00:32.577852 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.577695 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6e8ee8b7-3b67-4b9d-88fc-552fd655d11a-serviceca\") pod \"node-ca-54fff\" (UID: \"6e8ee8b7-3b67-4b9d-88fc-552fd655d11a\") " pod="openshift-image-registry/node-ca-54fff" Sep 11 20:00:32.577852 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.577740 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/50ff3d16-16c7-4631-bf85-6ee04ed3b508-etc-sysconfig\") pod \"tuned-f6skx\" (UID: \"50ff3d16-16c7-4631-bf85-6ee04ed3b508\") " pod="openshift-cluster-node-tuning-operator/tuned-f6skx" Sep 11 20:00:32.577852 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.577756 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/50ff3d16-16c7-4631-bf85-6ee04ed3b508-etc-modprobe-d\") pod \"tuned-f6skx\" (UID: \"50ff3d16-16c7-4631-bf85-6ee04ed3b508\") " pod="openshift-cluster-node-tuning-operator/tuned-f6skx" Sep 11 20:00:32.577852 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.577767 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce96d552-27b5-4912-86b0-38f0f46d0002-metrics-certs\") pod \"network-metrics-daemon-zpl5q\" (UID: \"ce96d552-27b5-4912-86b0-38f0f46d0002\") " pod="openshift-multus/network-metrics-daemon-zpl5q" Sep 11 20:00:32.577852 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.577788 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6e8ee8b7-3b67-4b9d-88fc-552fd655d11a-host\") pod \"node-ca-54fff\" (UID: \"6e8ee8b7-3b67-4b9d-88fc-552fd655d11a\") " pod="openshift-image-registry/node-ca-54fff" Sep 11 20:00:32.577852 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.577790 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/50ff3d16-16c7-4631-bf85-6ee04ed3b508-host\") pod \"tuned-f6skx\" (UID: \"50ff3d16-16c7-4631-bf85-6ee04ed3b508\") " pod="openshift-cluster-node-tuning-operator/tuned-f6skx" Sep 11 20:00:32.577852 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.577797 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/50ff3d16-16c7-4631-bf85-6ee04ed3b508-etc-sysconfig\") pod \"tuned-f6skx\" (UID: \"50ff3d16-16c7-4631-bf85-6ee04ed3b508\") " pod="openshift-cluster-node-tuning-operator/tuned-f6skx" Sep 11 20:00:32.577852 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.577831 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6e8ee8b7-3b67-4b9d-88fc-552fd655d11a-host\") pod \"node-ca-54fff\" (UID: \"6e8ee8b7-3b67-4b9d-88fc-552fd655d11a\") " pod="openshift-image-registry/node-ca-54fff" Sep 11 20:00:32.578245 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:32.577880 2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 20:00:32.578245 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.577885 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z872\" (UniqueName: \"kubernetes.io/projected/6e8ee8b7-3b67-4b9d-88fc-552fd655d11a-kube-api-access-2z872\") pod \"node-ca-54fff\" (UID: \"6e8ee8b7-3b67-4b9d-88fc-552fd655d11a\") " pod="openshift-image-registry/node-ca-54fff" Sep 11 20:00:32.578245 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:32.577951 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce96d552-27b5-4912-86b0-38f0f46d0002-metrics-certs podName:ce96d552-27b5-4912-86b0-38f0f46d0002 nodeName:}" failed. No retries permitted until 2026-09-11 20:00:33.077930765 +0000 UTC m=+3.125981106 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ce96d552-27b5-4912-86b0-38f0f46d0002-metrics-certs") pod "network-metrics-daemon-zpl5q" (UID: "ce96d552-27b5-4912-86b0-38f0f46d0002") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 20:00:32.578245 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.577987 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/50ff3d16-16c7-4631-bf85-6ee04ed3b508-etc-systemd\") pod \"tuned-f6skx\" (UID: \"50ff3d16-16c7-4631-bf85-6ee04ed3b508\") " pod="openshift-cluster-node-tuning-operator/tuned-f6skx" Sep 11 20:00:32.578245 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.578036 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50ff3d16-16c7-4631-bf85-6ee04ed3b508-sys\") pod \"tuned-f6skx\" (UID: \"50ff3d16-16c7-4631-bf85-6ee04ed3b508\") " pod="openshift-cluster-node-tuning-operator/tuned-f6skx" Sep 11 20:00:32.578245 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.578062 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/50ff3d16-16c7-4631-bf85-6ee04ed3b508-lib-modules\") pod \"tuned-f6skx\" (UID: \"50ff3d16-16c7-4631-bf85-6ee04ed3b508\") " pod="openshift-cluster-node-tuning-operator/tuned-f6skx" Sep 11 20:00:32.578245 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.578089 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b54lr\" (UniqueName: \"kubernetes.io/projected/ce96d552-27b5-4912-86b0-38f0f46d0002-kube-api-access-b54lr\") pod \"network-metrics-daemon-zpl5q\" (UID: \"ce96d552-27b5-4912-86b0-38f0f46d0002\") " pod="openshift-multus/network-metrics-daemon-zpl5q" Sep 11 20:00:32.578245 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.578113 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/482b4b76-dc42-490f-afcb-5b672806c329-iptables-alerter-script\") pod \"iptables-alerter-r64rv\" (UID: \"482b4b76-dc42-490f-afcb-5b672806c329\") " pod="openshift-network-operator/iptables-alerter-r64rv" Sep 11 20:00:32.578245 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.578114 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50ff3d16-16c7-4631-bf85-6ee04ed3b508-sys\") pod \"tuned-f6skx\" (UID: \"50ff3d16-16c7-4631-bf85-6ee04ed3b508\") " pod="openshift-cluster-node-tuning-operator/tuned-f6skx" Sep 11 20:00:32.578245 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.578121 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/50ff3d16-16c7-4631-bf85-6ee04ed3b508-etc-systemd\") pod \"tuned-f6skx\" (UID: \"50ff3d16-16c7-4631-bf85-6ee04ed3b508\") " pod="openshift-cluster-node-tuning-operator/tuned-f6skx" Sep 11 20:00:32.578245 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.578133 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc2r6\" (UniqueName: \"kubernetes.io/projected/482b4b76-dc42-490f-afcb-5b672806c329-kube-api-access-mc2r6\") pod \"iptables-alerter-r64rv\" (UID: \"482b4b76-dc42-490f-afcb-5b672806c329\") " pod="openshift-network-operator/iptables-alerter-r64rv" Sep 11 20:00:32.578245 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.578176 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/50ff3d16-16c7-4631-bf85-6ee04ed3b508-lib-modules\") pod \"tuned-f6skx\" (UID: \"50ff3d16-16c7-4631-bf85-6ee04ed3b508\") " pod="openshift-cluster-node-tuning-operator/tuned-f6skx" Sep 11 20:00:32.578245 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.578189 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6e8ee8b7-3b67-4b9d-88fc-552fd655d11a-serviceca\") pod \"node-ca-54fff\" (UID: \"6e8ee8b7-3b67-4b9d-88fc-552fd655d11a\") " pod="openshift-image-registry/node-ca-54fff" Sep 11 20:00:32.578245 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.578197 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1ab887bb-e9fb-4ad3-9178-a35ed6fecfed-konnectivity-ca\") pod \"konnectivity-agent-jdhk4\" (UID: \"1ab887bb-e9fb-4ad3-9178-a35ed6fecfed\") " pod="kube-system/konnectivity-agent-jdhk4" Sep 11 20:00:32.578245 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.578223 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/50ff3d16-16c7-4631-bf85-6ee04ed3b508-etc-sysctl-conf\") pod \"tuned-f6skx\" (UID: \"50ff3d16-16c7-4631-bf85-6ee04ed3b508\") " pod="openshift-cluster-node-tuning-operator/tuned-f6skx" Sep 11 20:00:32.578979 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.578254 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/50ff3d16-16c7-4631-bf85-6ee04ed3b508-run\") pod \"tuned-f6skx\" (UID: \"50ff3d16-16c7-4631-bf85-6ee04ed3b508\") " pod="openshift-cluster-node-tuning-operator/tuned-f6skx" Sep 11 20:00:32.578979 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.578279 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/482b4b76-dc42-490f-afcb-5b672806c329-host-slash\") pod \"iptables-alerter-r64rv\" (UID: \"482b4b76-dc42-490f-afcb-5b672806c329\") " pod="openshift-network-operator/iptables-alerter-r64rv" Sep 11 20:00:32.578979 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.578305 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1ab887bb-e9fb-4ad3-9178-a35ed6fecfed-agent-certs\") pod \"konnectivity-agent-jdhk4\" (UID: \"1ab887bb-e9fb-4ad3-9178-a35ed6fecfed\") " pod="kube-system/konnectivity-agent-jdhk4" Sep 11 20:00:32.578979 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.578329 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/50ff3d16-16c7-4631-bf85-6ee04ed3b508-etc-sysctl-d\") pod \"tuned-f6skx\" (UID: \"50ff3d16-16c7-4631-bf85-6ee04ed3b508\") " pod="openshift-cluster-node-tuning-operator/tuned-f6skx" Sep 11 20:00:32.578979 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.578349 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/50ff3d16-16c7-4631-bf85-6ee04ed3b508-etc-tuned\") pod \"tuned-f6skx\" (UID: \"50ff3d16-16c7-4631-bf85-6ee04ed3b508\") " pod="openshift-cluster-node-tuning-operator/tuned-f6skx" Sep 11 20:00:32.578979 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.578375 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/50ff3d16-16c7-4631-bf85-6ee04ed3b508-var-lib-kubelet\") pod \"tuned-f6skx\" (UID: \"50ff3d16-16c7-4631-bf85-6ee04ed3b508\") " pod="openshift-cluster-node-tuning-operator/tuned-f6skx" Sep 11 20:00:32.578979 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.578395 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/50ff3d16-16c7-4631-bf85-6ee04ed3b508-etc-sysctl-conf\") pod \"tuned-f6skx\" (UID: \"50ff3d16-16c7-4631-bf85-6ee04ed3b508\") " pod="openshift-cluster-node-tuning-operator/tuned-f6skx" Sep 11 20:00:32.578979 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.578403 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/482b4b76-dc42-490f-afcb-5b672806c329-host-slash\") pod \"iptables-alerter-r64rv\" (UID: \"482b4b76-dc42-490f-afcb-5b672806c329\") " pod="openshift-network-operator/iptables-alerter-r64rv" Sep 11 20:00:32.578979 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.578399 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/50ff3d16-16c7-4631-bf85-6ee04ed3b508-run\") pod \"tuned-f6skx\" (UID: \"50ff3d16-16c7-4631-bf85-6ee04ed3b508\") " pod="openshift-cluster-node-tuning-operator/tuned-f6skx" Sep 11 20:00:32.578979 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.578417 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/50ff3d16-16c7-4631-bf85-6ee04ed3b508-etc-kubernetes\") pod \"tuned-f6skx\" (UID: \"50ff3d16-16c7-4631-bf85-6ee04ed3b508\") " pod="openshift-cluster-node-tuning-operator/tuned-f6skx" Sep 11 20:00:32.578979 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.578441 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s645j\" (UniqueName: \"kubernetes.io/projected/50ff3d16-16c7-4631-bf85-6ee04ed3b508-kube-api-access-s645j\") pod \"tuned-f6skx\" (UID: \"50ff3d16-16c7-4631-bf85-6ee04ed3b508\") " pod="openshift-cluster-node-tuning-operator/tuned-f6skx" Sep 11 20:00:32.578979 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.578472 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50ff3d16-16c7-4631-bf85-6ee04ed3b508-tmp\") pod \"tuned-f6skx\" (UID: \"50ff3d16-16c7-4631-bf85-6ee04ed3b508\") " pod="openshift-cluster-node-tuning-operator/tuned-f6skx" Sep 11 20:00:32.578979 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.578503 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/50ff3d16-16c7-4631-bf85-6ee04ed3b508-etc-sysctl-d\") pod \"tuned-f6skx\" (UID: \"50ff3d16-16c7-4631-bf85-6ee04ed3b508\") " pod="openshift-cluster-node-tuning-operator/tuned-f6skx" Sep 11 20:00:32.578979 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.578554 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/50ff3d16-16c7-4631-bf85-6ee04ed3b508-etc-kubernetes\") pod \"tuned-f6skx\" (UID: \"50ff3d16-16c7-4631-bf85-6ee04ed3b508\") " pod="openshift-cluster-node-tuning-operator/tuned-f6skx" Sep 11 20:00:32.578979 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.578619 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/50ff3d16-16c7-4631-bf85-6ee04ed3b508-var-lib-kubelet\") pod \"tuned-f6skx\" (UID: \"50ff3d16-16c7-4631-bf85-6ee04ed3b508\") " pod="openshift-cluster-node-tuning-operator/tuned-f6skx" Sep 11 20:00:32.578979 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.578699 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/482b4b76-dc42-490f-afcb-5b672806c329-iptables-alerter-script\") pod \"iptables-alerter-r64rv\" (UID: \"482b4b76-dc42-490f-afcb-5b672806c329\") " pod="openshift-network-operator/iptables-alerter-r64rv" Sep 11 20:00:32.578979 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.578781 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1ab887bb-e9fb-4ad3-9178-a35ed6fecfed-konnectivity-ca\") pod \"konnectivity-agent-jdhk4\" (UID: \"1ab887bb-e9fb-4ad3-9178-a35ed6fecfed\") " pod="kube-system/konnectivity-agent-jdhk4" Sep 11 20:00:32.580959 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.580893 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/50ff3d16-16c7-4631-bf85-6ee04ed3b508-etc-tuned\") pod \"tuned-f6skx\" (UID: \"50ff3d16-16c7-4631-bf85-6ee04ed3b508\") " pod="openshift-cluster-node-tuning-operator/tuned-f6skx" Sep 11 20:00:32.581121 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.581103 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1ab887bb-e9fb-4ad3-9178-a35ed6fecfed-agent-certs\") pod \"konnectivity-agent-jdhk4\" (UID: \"1ab887bb-e9fb-4ad3-9178-a35ed6fecfed\") " pod="kube-system/konnectivity-agent-jdhk4" Sep 11 20:00:32.581297 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.581276 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50ff3d16-16c7-4631-bf85-6ee04ed3b508-tmp\") pod \"tuned-f6skx\" (UID: \"50ff3d16-16c7-4631-bf85-6ee04ed3b508\") " pod="openshift-cluster-node-tuning-operator/tuned-f6skx" Sep 11 20:00:32.587019 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.586976 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z872\" (UniqueName: \"kubernetes.io/projected/6e8ee8b7-3b67-4b9d-88fc-552fd655d11a-kube-api-access-2z872\") pod \"node-ca-54fff\" (UID: \"6e8ee8b7-3b67-4b9d-88fc-552fd655d11a\") " pod="openshift-image-registry/node-ca-54fff" Sep 11 20:00:32.587841 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.587815 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc2r6\" (UniqueName: \"kubernetes.io/projected/482b4b76-dc42-490f-afcb-5b672806c329-kube-api-access-mc2r6\") pod \"iptables-alerter-r64rv\" (UID: \"482b4b76-dc42-490f-afcb-5b672806c329\") " pod="openshift-network-operator/iptables-alerter-r64rv" Sep 11 20:00:32.588157 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.588138 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s645j\" (UniqueName: \"kubernetes.io/projected/50ff3d16-16c7-4631-bf85-6ee04ed3b508-kube-api-access-s645j\") pod \"tuned-f6skx\" (UID: \"50ff3d16-16c7-4631-bf85-6ee04ed3b508\") " pod="openshift-cluster-node-tuning-operator/tuned-f6skx" Sep 11 20:00:32.588380 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.588363 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b54lr\" (UniqueName: \"kubernetes.io/projected/ce96d552-27b5-4912-86b0-38f0f46d0002-kube-api-access-b54lr\") pod \"network-metrics-daemon-zpl5q\" (UID: \"ce96d552-27b5-4912-86b0-38f0f46d0002\") " pod="openshift-multus/network-metrics-daemon-zpl5q" Sep 11 20:00:32.657551 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.657522 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2m6cs" Sep 11 20:00:32.664857 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.664833 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w6p5s" Sep 11 20:00:32.675551 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.675531 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8kcqg" Sep 11 20:00:32.679356 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.679338 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ddmz9" Sep 11 20:00:32.687052 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.687035 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:32.694053 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.694035 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-54fff" Sep 11 20:00:32.701640 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.701618 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jdhk4" Sep 11 20:00:32.710266 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.710246 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-r64rv" Sep 11 20:00:32.713877 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.713860 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-f6skx" Sep 11 20:00:32.831477 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.831391 2546 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 20:00:32.981500 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.981464 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1f86594f-112f-4e58-a2ca-bbd951c4d1af-original-pull-secret\") pod \"global-pull-secret-syncer-q2dm2\" (UID: \"1f86594f-112f-4e58-a2ca-bbd951c4d1af\") " pod="kube-system/global-pull-secret-syncer-q2dm2" Sep 11 20:00:32.981673 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:32.981507 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwhtf\" (UniqueName: \"kubernetes.io/projected/ae2b5d1f-4bb1-41eb-9096-d54b304ad672-kube-api-access-cwhtf\") pod \"network-check-target-wk9hk\" (UID: \"ae2b5d1f-4bb1-41eb-9096-d54b304ad672\") " pod="openshift-network-diagnostics/network-check-target-wk9hk" Sep 11 20:00:32.981673 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:32.981628 2546 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 20:00:32.981673 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:32.981642 2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 20:00:32.981673 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:32.981660 2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 20:00:32.981673 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:32.981670 2546 projected.go:196] Error preparing data for projected volume kube-api-access-cwhtf for pod openshift-network-diagnostics/network-check-target-wk9hk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 20:00:32.981946 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:32.981688 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f86594f-112f-4e58-a2ca-bbd951c4d1af-original-pull-secret podName:1f86594f-112f-4e58-a2ca-bbd951c4d1af nodeName:}" failed. No retries permitted until 2026-09-11 20:00:33.981671679 +0000 UTC m=+4.029722001 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1f86594f-112f-4e58-a2ca-bbd951c4d1af-original-pull-secret") pod "global-pull-secret-syncer-q2dm2" (UID: "1f86594f-112f-4e58-a2ca-bbd951c4d1af") : object "kube-system"/"original-pull-secret" not registered Sep 11 20:00:32.981946 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:32.981710 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae2b5d1f-4bb1-41eb-9096-d54b304ad672-kube-api-access-cwhtf podName:ae2b5d1f-4bb1-41eb-9096-d54b304ad672 nodeName:}" failed. No retries permitted until 2026-09-11 20:00:33.981696002 +0000 UTC m=+4.029746337 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-cwhtf" (UniqueName: "kubernetes.io/projected/ae2b5d1f-4bb1-41eb-9096-d54b304ad672-kube-api-access-cwhtf") pod "network-check-target-wk9hk" (UID: "ae2b5d1f-4bb1-41eb-9096-d54b304ad672") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 20:00:33.081895 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:33.081824 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce96d552-27b5-4912-86b0-38f0f46d0002-metrics-certs\") pod \"network-metrics-daemon-zpl5q\" (UID: \"ce96d552-27b5-4912-86b0-38f0f46d0002\") " pod="openshift-multus/network-metrics-daemon-zpl5q" Sep 11 20:00:33.082047 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:33.081976 2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 20:00:33.082047 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:33.082036 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce96d552-27b5-4912-86b0-38f0f46d0002-metrics-certs podName:ce96d552-27b5-4912-86b0-38f0f46d0002 nodeName:}" failed. No retries permitted until 2026-09-11 20:00:34.082021409 +0000 UTC m=+4.130071731 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ce96d552-27b5-4912-86b0-38f0f46d0002-metrics-certs") pod "network-metrics-daemon-zpl5q" (UID: "ce96d552-27b5-4912-86b0-38f0f46d0002") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 20:00:33.404559 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:33.404464 2546 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 19:55:32 +0000 UTC" deadline="2028-04-24 17:04:26.174605683 +0000 UTC" Sep 11 20:00:33.404559 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:33.404507 2546 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14181h3m52.770106363s" Sep 11 20:00:33.648103 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:33.648073 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod171c51ac_8f0e_426f_973d_5a596e73d7d1.slice/crio-eb95e206575f6beb70500b8893603d7ba0d384196a7e646ed896f2407afb4316 WatchSource:0}: Error finding container eb95e206575f6beb70500b8893603d7ba0d384196a7e646ed896f2407afb4316: Status 404 returned error can't find the container with id eb95e206575f6beb70500b8893603d7ba0d384196a7e646ed896f2407afb4316 Sep 11 20:00:33.650178 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:33.650036 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e8ee8b7_3b67_4b9d_88fc_552fd655d11a.slice/crio-b3ad23ae34b06efc4dae60c99b1737801dac626c1f8d64c09b837b4ca12672f9 WatchSource:0}: Error finding container b3ad23ae34b06efc4dae60c99b1737801dac626c1f8d64c09b837b4ca12672f9: Status 404 returned error can't find the container with id b3ad23ae34b06efc4dae60c99b1737801dac626c1f8d64c09b837b4ca12672f9 Sep 11 20:00:33.652742 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:33.652722 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6b30385_b0e1_4ab5_916a_57992046f196.slice/crio-63b0ae5bbbf38bde245081fbc3b2a8516f56a6ec3a3e6b4dd6c959032ca9cd24 WatchSource:0}: Error finding container 63b0ae5bbbf38bde245081fbc3b2a8516f56a6ec3a3e6b4dd6c959032ca9cd24: Status 404 returned error can't find the container with id 63b0ae5bbbf38bde245081fbc3b2a8516f56a6ec3a3e6b4dd6c959032ca9cd24 Sep 11 20:00:33.654192 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:33.654117 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod482b4b76_dc42_490f_afcb_5b672806c329.slice/crio-ef1a5d950ff663f836b7af6890ffba42541d649eb97b94e59ccae55e1033be98 WatchSource:0}: Error finding container ef1a5d950ff663f836b7af6890ffba42541d649eb97b94e59ccae55e1033be98: Status 404 returned error can't find the container with id ef1a5d950ff663f836b7af6890ffba42541d649eb97b94e59ccae55e1033be98 Sep 11 20:00:33.655055 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:33.655028 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2133a2c0_f263_45aa_9d96_1819721a8c9b.slice/crio-264e24eea141046ab3c29a606e819ebbc4ceae5808a5021a84e411d40142c28c WatchSource:0}: Error finding container 264e24eea141046ab3c29a606e819ebbc4ceae5808a5021a84e411d40142c28c: Status 404 returned error can't find the container with id 264e24eea141046ab3c29a606e819ebbc4ceae5808a5021a84e411d40142c28c Sep 11 20:00:33.658252 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:00:33.657773 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52c72e1a_d061_4d3c_922b_b572a6496fed.slice/crio-7cdecf4dae391fc474a9e98bfdc8e6fe0980e517597ab5bfcf0a1562507d0780 WatchSource:0}: Error finding container 7cdecf4dae391fc474a9e98bfdc8e6fe0980e517597ab5bfcf0a1562507d0780: Status 404 returned error can't find the container with id 7cdecf4dae391fc474a9e98bfdc8e6fe0980e517597ab5bfcf0a1562507d0780 Sep 11 20:00:33.989815 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:33.989781 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1f86594f-112f-4e58-a2ca-bbd951c4d1af-original-pull-secret\") pod \"global-pull-secret-syncer-q2dm2\" (UID: \"1f86594f-112f-4e58-a2ca-bbd951c4d1af\") " pod="kube-system/global-pull-secret-syncer-q2dm2" Sep 11 20:00:33.989950 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:33.989823 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwhtf\" (UniqueName: \"kubernetes.io/projected/ae2b5d1f-4bb1-41eb-9096-d54b304ad672-kube-api-access-cwhtf\") pod \"network-check-target-wk9hk\" (UID: \"ae2b5d1f-4bb1-41eb-9096-d54b304ad672\") " pod="openshift-network-diagnostics/network-check-target-wk9hk" Sep 11 20:00:33.989950 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:33.989917 2546 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 20:00:33.989950 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:33.989939 2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 20:00:33.990080 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:33.989956 2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 20:00:33.990080 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:33.989968 2546 projected.go:196] Error preparing data for projected volume kube-api-access-cwhtf for pod openshift-network-diagnostics/network-check-target-wk9hk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 20:00:33.990080 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:33.989974 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f86594f-112f-4e58-a2ca-bbd951c4d1af-original-pull-secret podName:1f86594f-112f-4e58-a2ca-bbd951c4d1af nodeName:}" failed. No retries permitted until 2026-09-11 20:00:35.989961244 +0000 UTC m=+6.038011587 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1f86594f-112f-4e58-a2ca-bbd951c4d1af-original-pull-secret") pod "global-pull-secret-syncer-q2dm2" (UID: "1f86594f-112f-4e58-a2ca-bbd951c4d1af") : object "kube-system"/"original-pull-secret" not registered Sep 11 20:00:33.990080 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:33.990013 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae2b5d1f-4bb1-41eb-9096-d54b304ad672-kube-api-access-cwhtf podName:ae2b5d1f-4bb1-41eb-9096-d54b304ad672 nodeName:}" failed. No retries permitted until 2026-09-11 20:00:35.989998059 +0000 UTC m=+6.038048398 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-cwhtf" (UniqueName: "kubernetes.io/projected/ae2b5d1f-4bb1-41eb-9096-d54b304ad672-kube-api-access-cwhtf") pod "network-check-target-wk9hk" (UID: "ae2b5d1f-4bb1-41eb-9096-d54b304ad672") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 20:00:34.090953 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:34.090920 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce96d552-27b5-4912-86b0-38f0f46d0002-metrics-certs\") pod \"network-metrics-daemon-zpl5q\" (UID: \"ce96d552-27b5-4912-86b0-38f0f46d0002\") " pod="openshift-multus/network-metrics-daemon-zpl5q" Sep 11 20:00:34.091088 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:34.091041 2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 20:00:34.091148 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:34.091096 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce96d552-27b5-4912-86b0-38f0f46d0002-metrics-certs podName:ce96d552-27b5-4912-86b0-38f0f46d0002 nodeName:}" failed. No retries permitted until 2026-09-11 20:00:36.091083734 +0000 UTC m=+6.139134069 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ce96d552-27b5-4912-86b0-38f0f46d0002-metrics-certs") pod "network-metrics-daemon-zpl5q" (UID: "ce96d552-27b5-4912-86b0-38f0f46d0002") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 20:00:34.405012 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:34.404872 2546 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 19:55:32 +0000 UTC" deadline="2028-04-10 19:51:33.526854801 +0000 UTC" Sep 11 20:00:34.405012 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:34.404934 2546 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13847h50m59.121925641s" Sep 11 20:00:34.446133 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:34.445692 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zpl5q" Sep 11 20:00:34.446133 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:34.445702 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wk9hk" Sep 11 20:00:34.446133 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:34.445825 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2dm2" Sep 11 20:00:34.446133 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:34.445825 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zpl5q" podUID="ce96d552-27b5-4912-86b0-38f0f46d0002" Sep 11 20:00:34.446133 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:34.445952 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wk9hk" podUID="ae2b5d1f-4bb1-41eb-9096-d54b304ad672" Sep 11 20:00:34.446133 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:34.446031 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2dm2" podUID="1f86594f-112f-4e58-a2ca-bbd951c4d1af" Sep 11 20:00:34.464011 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:34.463870 2546 generic.go:356] "Generic (PLEG): container finished" podID="d907173ce62f903e18094a4e7845248d" containerID="f720f4b2f3a7f70b3e66fec63a4df83886f4b25bb46b010c8923c98dcc92b101" exitCode=0 Sep 11 20:00:34.464011 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:34.463958 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-205.ec2.internal" event={"ID":"d907173ce62f903e18094a4e7845248d","Type":"ContainerDied","Data":"f720f4b2f3a7f70b3e66fec63a4df83886f4b25bb46b010c8923c98dcc92b101"} Sep 11 20:00:34.470513 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:34.470456 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-205.ec2.internal" event={"ID":"11ecaef2c363c5ae25ba196bd5dc1142","Type":"ContainerStarted","Data":"591efd0d495b5882b324111ceb385250a85dd44cbfcd7b7a1d0e873bf60fcac5"} Sep 11 20:00:34.474809 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:34.474734 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-r64rv" event={"ID":"482b4b76-dc42-490f-afcb-5b672806c329","Type":"ContainerStarted","Data":"ef1a5d950ff663f836b7af6890ffba42541d649eb97b94e59ccae55e1033be98"} Sep 11 20:00:34.480129 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:34.480058 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-f6skx" event={"ID":"50ff3d16-16c7-4631-bf85-6ee04ed3b508","Type":"ContainerStarted","Data":"8d23293b740aaf3a301d57b4ffe156b0e42c4a3813b97916c230ae2580f8bd56"} Sep 11 20:00:34.482990 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:34.482933 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ddmz9" event={"ID":"52c72e1a-d061-4d3c-922b-b572a6496fed","Type":"ContainerStarted","Data":"7cdecf4dae391fc474a9e98bfdc8e6fe0980e517597ab5bfcf0a1562507d0780"} Sep 11 20:00:34.484478 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:34.484416 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2m6cs" event={"ID":"2133a2c0-f263-45aa-9d96-1819721a8c9b","Type":"ContainerStarted","Data":"264e24eea141046ab3c29a606e819ebbc4ceae5808a5021a84e411d40142c28c"} Sep 11 20:00:34.486804 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:34.486708 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" event={"ID":"b6b30385-b0e1-4ab5-916a-57992046f196","Type":"ContainerStarted","Data":"63b0ae5bbbf38bde245081fbc3b2a8516f56a6ec3a3e6b4dd6c959032ca9cd24"} Sep 11 20:00:34.498169 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:34.498094 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-54fff" event={"ID":"6e8ee8b7-3b67-4b9d-88fc-552fd655d11a","Type":"ContainerStarted","Data":"b3ad23ae34b06efc4dae60c99b1737801dac626c1f8d64c09b837b4ca12672f9"} Sep 11 20:00:34.504667 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:34.504636 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jdhk4" event={"ID":"1ab887bb-e9fb-4ad3-9178-a35ed6fecfed","Type":"ContainerStarted","Data":"0bef8f4f3e5ffc76879b328b682e357ad882a9000aa61bb4dd65d4cf075ccd7c"} Sep 11 20:00:34.509913 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:34.509825 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w6p5s" event={"ID":"52601b9c-beef-4b1c-aaf3-34cf267a00fe","Type":"ContainerStarted","Data":"87c99319521045dfe057cc19430ff44da7716a11c5d9a0d90b32bfcdb0f67d48"} Sep 11 20:00:34.521587 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:34.521542 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8kcqg" event={"ID":"171c51ac-8f0e-426f-973d-5a596e73d7d1","Type":"ContainerStarted","Data":"eb95e206575f6beb70500b8893603d7ba0d384196a7e646ed896f2407afb4316"} Sep 11 20:00:35.536711 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:35.536633 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-205.ec2.internal" event={"ID":"d907173ce62f903e18094a4e7845248d","Type":"ContainerStarted","Data":"7c1b5f807a9f7a8d8355c8e6ac4f78cf3676185ae36d506c24fd84b2b11f066b"} Sep 11 20:00:35.549877 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:35.549824 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-205.ec2.internal" podStartSLOduration=4.549805072 podStartE2EDuration="4.549805072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 20:00:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:00:34.496565181 +0000 UTC m=+4.544615522" watchObservedRunningTime="2026-09-11 20:00:35.549805072 +0000 UTC m=+5.597855417" Sep 11 20:00:36.009767 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:36.008850 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1f86594f-112f-4e58-a2ca-bbd951c4d1af-original-pull-secret\") pod \"global-pull-secret-syncer-q2dm2\" (UID: \"1f86594f-112f-4e58-a2ca-bbd951c4d1af\") " pod="kube-system/global-pull-secret-syncer-q2dm2" Sep 11 20:00:36.009767 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:36.008901 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwhtf\" (UniqueName: \"kubernetes.io/projected/ae2b5d1f-4bb1-41eb-9096-d54b304ad672-kube-api-access-cwhtf\") pod \"network-check-target-wk9hk\" (UID: \"ae2b5d1f-4bb1-41eb-9096-d54b304ad672\") " pod="openshift-network-diagnostics/network-check-target-wk9hk" Sep 11 20:00:36.009767 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:36.009052 2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 20:00:36.009767 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:36.009070 2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 20:00:36.009767 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:36.009082 2546 projected.go:196] Error preparing data for projected volume kube-api-access-cwhtf for pod openshift-network-diagnostics/network-check-target-wk9hk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 20:00:36.009767 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:36.009137 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae2b5d1f-4bb1-41eb-9096-d54b304ad672-kube-api-access-cwhtf podName:ae2b5d1f-4bb1-41eb-9096-d54b304ad672 nodeName:}" failed. No retries permitted until 2026-09-11 20:00:40.009119201 +0000 UTC m=+10.057169540 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-cwhtf" (UniqueName: "kubernetes.io/projected/ae2b5d1f-4bb1-41eb-9096-d54b304ad672-kube-api-access-cwhtf") pod "network-check-target-wk9hk" (UID: "ae2b5d1f-4bb1-41eb-9096-d54b304ad672") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 20:00:36.009767 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:36.009624 2546 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 20:00:36.009767 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:36.009674 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f86594f-112f-4e58-a2ca-bbd951c4d1af-original-pull-secret podName:1f86594f-112f-4e58-a2ca-bbd951c4d1af nodeName:}" failed. No retries permitted until 2026-09-11 20:00:40.009660114 +0000 UTC m=+10.057710451 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1f86594f-112f-4e58-a2ca-bbd951c4d1af-original-pull-secret") pod "global-pull-secret-syncer-q2dm2" (UID: "1f86594f-112f-4e58-a2ca-bbd951c4d1af") : object "kube-system"/"original-pull-secret" not registered Sep 11 20:00:36.109742 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:36.109707 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce96d552-27b5-4912-86b0-38f0f46d0002-metrics-certs\") pod \"network-metrics-daemon-zpl5q\" (UID: \"ce96d552-27b5-4912-86b0-38f0f46d0002\") " pod="openshift-multus/network-metrics-daemon-zpl5q" Sep 11 20:00:36.109905 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:36.109863 2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 20:00:36.109976 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:36.109922 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce96d552-27b5-4912-86b0-38f0f46d0002-metrics-certs podName:ce96d552-27b5-4912-86b0-38f0f46d0002 nodeName:}" failed. No retries permitted until 2026-09-11 20:00:40.109905106 +0000 UTC m=+10.157955433 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ce96d552-27b5-4912-86b0-38f0f46d0002-metrics-certs") pod "network-metrics-daemon-zpl5q" (UID: "ce96d552-27b5-4912-86b0-38f0f46d0002") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 20:00:36.445616 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:36.445502 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wk9hk" Sep 11 20:00:36.445774 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:36.445665 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wk9hk" podUID="ae2b5d1f-4bb1-41eb-9096-d54b304ad672" Sep 11 20:00:36.446070 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:36.446050 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2dm2" Sep 11 20:00:36.446183 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:36.446141 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2dm2" podUID="1f86594f-112f-4e58-a2ca-bbd951c4d1af" Sep 11 20:00:36.446252 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:36.446206 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zpl5q" Sep 11 20:00:36.446346 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:36.446315 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zpl5q" podUID="ce96d552-27b5-4912-86b0-38f0f46d0002" Sep 11 20:00:38.445745 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:38.445695 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wk9hk" Sep 11 20:00:38.446212 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:38.445821 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wk9hk" podUID="ae2b5d1f-4bb1-41eb-9096-d54b304ad672" Sep 11 20:00:38.446288 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:38.446217 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2dm2" Sep 11 20:00:38.446349 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:38.446316 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2dm2" podUID="1f86594f-112f-4e58-a2ca-bbd951c4d1af" Sep 11 20:00:38.446401 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:38.446380 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zpl5q" Sep 11 20:00:38.446484 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:38.446464 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zpl5q" podUID="ce96d552-27b5-4912-86b0-38f0f46d0002" Sep 11 20:00:40.043779 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:40.043527 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1f86594f-112f-4e58-a2ca-bbd951c4d1af-original-pull-secret\") pod \"global-pull-secret-syncer-q2dm2\" (UID: \"1f86594f-112f-4e58-a2ca-bbd951c4d1af\") " pod="kube-system/global-pull-secret-syncer-q2dm2" Sep 11 20:00:40.044248 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:40.043808 2546 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 20:00:40.044248 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:40.043885 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f86594f-112f-4e58-a2ca-bbd951c4d1af-original-pull-secret podName:1f86594f-112f-4e58-a2ca-bbd951c4d1af nodeName:}" failed. No retries permitted until 2026-09-11 20:00:48.043865258 +0000 UTC m=+18.091915584 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1f86594f-112f-4e58-a2ca-bbd951c4d1af-original-pull-secret") pod "global-pull-secret-syncer-q2dm2" (UID: "1f86594f-112f-4e58-a2ca-bbd951c4d1af") : object "kube-system"/"original-pull-secret" not registered Sep 11 20:00:40.044248 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:40.043812 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwhtf\" (UniqueName: \"kubernetes.io/projected/ae2b5d1f-4bb1-41eb-9096-d54b304ad672-kube-api-access-cwhtf\") pod \"network-check-target-wk9hk\" (UID: \"ae2b5d1f-4bb1-41eb-9096-d54b304ad672\") " pod="openshift-network-diagnostics/network-check-target-wk9hk" Sep 11 20:00:40.044248 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:40.043921 2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 20:00:40.044248 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:40.043945 2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 20:00:40.044248 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:40.043958 2546 projected.go:196] Error preparing data for projected volume kube-api-access-cwhtf for pod openshift-network-diagnostics/network-check-target-wk9hk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 20:00:40.044248 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:40.044008 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae2b5d1f-4bb1-41eb-9096-d54b304ad672-kube-api-access-cwhtf podName:ae2b5d1f-4bb1-41eb-9096-d54b304ad672 nodeName:}" failed. No retries permitted until 2026-09-11 20:00:48.043992676 +0000 UTC m=+18.092043000 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-cwhtf" (UniqueName: "kubernetes.io/projected/ae2b5d1f-4bb1-41eb-9096-d54b304ad672-kube-api-access-cwhtf") pod "network-check-target-wk9hk" (UID: "ae2b5d1f-4bb1-41eb-9096-d54b304ad672") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 20:00:40.145326 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:40.144723 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce96d552-27b5-4912-86b0-38f0f46d0002-metrics-certs\") pod \"network-metrics-daemon-zpl5q\" (UID: \"ce96d552-27b5-4912-86b0-38f0f46d0002\") " pod="openshift-multus/network-metrics-daemon-zpl5q" Sep 11 20:00:40.145326 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:40.144918 2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 20:00:40.145326 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:40.144979 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce96d552-27b5-4912-86b0-38f0f46d0002-metrics-certs podName:ce96d552-27b5-4912-86b0-38f0f46d0002 nodeName:}" failed. No retries permitted until 2026-09-11 20:00:48.144962509 +0000 UTC m=+18.193012844 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ce96d552-27b5-4912-86b0-38f0f46d0002-metrics-certs") pod "network-metrics-daemon-zpl5q" (UID: "ce96d552-27b5-4912-86b0-38f0f46d0002") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 20:00:40.444869 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:40.444792 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2dm2" Sep 11 20:00:40.445020 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:40.444933 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2dm2" podUID="1f86594f-112f-4e58-a2ca-bbd951c4d1af" Sep 11 20:00:40.445327 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:40.445307 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zpl5q" Sep 11 20:00:40.445465 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:40.445441 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zpl5q" podUID="ce96d552-27b5-4912-86b0-38f0f46d0002" Sep 11 20:00:40.445541 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:40.445496 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wk9hk" Sep 11 20:00:40.445618 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:40.445569 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wk9hk" podUID="ae2b5d1f-4bb1-41eb-9096-d54b304ad672" Sep 11 20:00:42.444388 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:42.444350 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2dm2" Sep 11 20:00:42.444865 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:42.444401 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zpl5q" Sep 11 20:00:42.444865 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:42.444350 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wk9hk" Sep 11 20:00:42.444865 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:42.444486 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2dm2" podUID="1f86594f-112f-4e58-a2ca-bbd951c4d1af" Sep 11 20:00:42.444865 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:42.444554 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wk9hk" podUID="ae2b5d1f-4bb1-41eb-9096-d54b304ad672" Sep 11 20:00:42.444865 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:42.444664 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zpl5q" podUID="ce96d552-27b5-4912-86b0-38f0f46d0002" Sep 11 20:00:44.444279 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:44.444189 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wk9hk" Sep 11 20:00:44.444279 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:44.444216 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zpl5q" Sep 11 20:00:44.444771 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:44.444198 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2dm2" Sep 11 20:00:44.444771 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:44.444324 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wk9hk" podUID="ae2b5d1f-4bb1-41eb-9096-d54b304ad672" Sep 11 20:00:44.444771 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:44.444447 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2dm2" podUID="1f86594f-112f-4e58-a2ca-bbd951c4d1af" Sep 11 20:00:44.444771 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:44.444532 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zpl5q" podUID="ce96d552-27b5-4912-86b0-38f0f46d0002" Sep 11 20:00:46.443692 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:46.443640 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2dm2" Sep 11 20:00:46.443692 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:46.443661 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zpl5q" Sep 11 20:00:46.443692 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:46.443660 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wk9hk" Sep 11 20:00:46.444253 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:46.443764 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2dm2" podUID="1f86594f-112f-4e58-a2ca-bbd951c4d1af" Sep 11 20:00:46.444253 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:46.443862 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zpl5q" podUID="ce96d552-27b5-4912-86b0-38f0f46d0002" Sep 11 20:00:46.444371 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:46.444275 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wk9hk" podUID="ae2b5d1f-4bb1-41eb-9096-d54b304ad672" Sep 11 20:00:48.104811 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:48.104722 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1f86594f-112f-4e58-a2ca-bbd951c4d1af-original-pull-secret\") pod \"global-pull-secret-syncer-q2dm2\" (UID: \"1f86594f-112f-4e58-a2ca-bbd951c4d1af\") " pod="kube-system/global-pull-secret-syncer-q2dm2" Sep 11 20:00:48.104811 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:48.104763 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwhtf\" (UniqueName: \"kubernetes.io/projected/ae2b5d1f-4bb1-41eb-9096-d54b304ad672-kube-api-access-cwhtf\") pod \"network-check-target-wk9hk\" (UID: \"ae2b5d1f-4bb1-41eb-9096-d54b304ad672\") " pod="openshift-network-diagnostics/network-check-target-wk9hk" Sep 11 20:00:48.105293 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:48.104891 2546 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 20:00:48.105293 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:48.104905 2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 20:00:48.105293 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:48.104927 2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 20:00:48.105293 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:48.104941 2546 projected.go:196] Error preparing data for projected volume kube-api-access-cwhtf for pod openshift-network-diagnostics/network-check-target-wk9hk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 20:00:48.105293 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:48.104987 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f86594f-112f-4e58-a2ca-bbd951c4d1af-original-pull-secret podName:1f86594f-112f-4e58-a2ca-bbd951c4d1af nodeName:}" failed. No retries permitted until 2026-09-11 20:01:04.10496459 +0000 UTC m=+34.153014925 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1f86594f-112f-4e58-a2ca-bbd951c4d1af-original-pull-secret") pod "global-pull-secret-syncer-q2dm2" (UID: "1f86594f-112f-4e58-a2ca-bbd951c4d1af") : object "kube-system"/"original-pull-secret" not registered Sep 11 20:00:48.105293 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:48.105007 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae2b5d1f-4bb1-41eb-9096-d54b304ad672-kube-api-access-cwhtf podName:ae2b5d1f-4bb1-41eb-9096-d54b304ad672 nodeName:}" failed. No retries permitted until 2026-09-11 20:01:04.104996674 +0000 UTC m=+34.153047009 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-cwhtf" (UniqueName: "kubernetes.io/projected/ae2b5d1f-4bb1-41eb-9096-d54b304ad672-kube-api-access-cwhtf") pod "network-check-target-wk9hk" (UID: "ae2b5d1f-4bb1-41eb-9096-d54b304ad672") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 20:00:48.205519 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:48.205476 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce96d552-27b5-4912-86b0-38f0f46d0002-metrics-certs\") pod \"network-metrics-daemon-zpl5q\" (UID: \"ce96d552-27b5-4912-86b0-38f0f46d0002\") " pod="openshift-multus/network-metrics-daemon-zpl5q" Sep 11 20:00:48.205696 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:48.205638 2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 20:00:48.205752 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:48.205700 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce96d552-27b5-4912-86b0-38f0f46d0002-metrics-certs podName:ce96d552-27b5-4912-86b0-38f0f46d0002 nodeName:}" failed. No retries permitted until 2026-09-11 20:01:04.205685456 +0000 UTC m=+34.253735778 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ce96d552-27b5-4912-86b0-38f0f46d0002-metrics-certs") pod "network-metrics-daemon-zpl5q" (UID: "ce96d552-27b5-4912-86b0-38f0f46d0002") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 20:00:48.444036 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:48.443944 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2dm2" Sep 11 20:00:48.444202 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:48.443959 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zpl5q" Sep 11 20:00:48.444202 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:48.444076 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2dm2" podUID="1f86594f-112f-4e58-a2ca-bbd951c4d1af" Sep 11 20:00:48.444202 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:48.443963 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wk9hk" Sep 11 20:00:48.444202 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:48.444183 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zpl5q" podUID="ce96d552-27b5-4912-86b0-38f0f46d0002" Sep 11 20:00:48.444395 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:48.444222 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wk9hk" podUID="ae2b5d1f-4bb1-41eb-9096-d54b304ad672" Sep 11 20:00:50.445499 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:50.445465 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zpl5q" Sep 11 20:00:50.446072 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:50.445552 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wk9hk" Sep 11 20:00:50.446135 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:50.445603 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2dm2" Sep 11 20:00:50.446208 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:50.446176 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wk9hk" podUID="ae2b5d1f-4bb1-41eb-9096-d54b304ad672" Sep 11 20:00:50.446321 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:50.446204 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zpl5q" podUID="ce96d552-27b5-4912-86b0-38f0f46d0002" Sep 11 20:00:50.446321 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:50.446228 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2dm2" podUID="1f86594f-112f-4e58-a2ca-bbd951c4d1af" Sep 11 20:00:51.573373 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:51.573338 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-f6skx" event={"ID":"50ff3d16-16c7-4631-bf85-6ee04ed3b508","Type":"ContainerStarted","Data":"bdd29a383f0527e035e4265f98ff1a30e7299d6874ab6be9c95f39fa04ee547f"} Sep 11 20:00:51.574844 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:51.574818 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ddmz9" event={"ID":"52c72e1a-d061-4d3c-922b-b572a6496fed","Type":"ContainerStarted","Data":"ef5842bbc5b587b4f43d4c3d85073aba6c0f2840d0c3132b713a00d8c971b146"} Sep 11 20:00:51.576462 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:51.576410 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2m6cs" event={"ID":"2133a2c0-f263-45aa-9d96-1819721a8c9b","Type":"ContainerStarted","Data":"cbcceffcd0f3b769c44af440935a9c57dd44806c547f294bb823c3ba38706221"} Sep 11 20:00:51.580061 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:51.579834 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" event={"ID":"b6b30385-b0e1-4ab5-916a-57992046f196","Type":"ContainerStarted","Data":"5b140d34175a023bba4a6bd442beaf42704e4bfb1799ee8c8c2a3f3a155c0137"} Sep 11 20:00:51.581533 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:51.581473 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jdhk4" event={"ID":"1ab887bb-e9fb-4ad3-9178-a35ed6fecfed","Type":"ContainerStarted","Data":"2b93195fb16de0f235ac71a3e22620e90bc75e89dffd5364a4596bc748268d97"} Sep 11 20:00:51.584108 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:51.584065 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w6p5s" event={"ID":"52601b9c-beef-4b1c-aaf3-34cf267a00fe","Type":"ContainerStarted","Data":"375653ec5169aa1ee5fd6902649b75477abdfb5dddf7ee7c539a862d341da347"} Sep 11 20:00:51.586385 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:51.586354 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8kcqg" event={"ID":"171c51ac-8f0e-426f-973d-5a596e73d7d1","Type":"ContainerStarted","Data":"7d00ca2cdd598216aad6f559e516b6522d1513f0e65feeb6e962fe0382edf997"} Sep 11 20:00:51.595665 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:51.595612 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-f6skx" podStartSLOduration=3.900744592 podStartE2EDuration="21.595597689s" totalImagesPullingTime="17.694853097s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 20:00:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:00:51.595529388 +0000 UTC m=+21.643579732" watchObservedRunningTime="2026-09-11 20:00:51.595597689 +0000 UTC m=+21.643648027" Sep 11 20:00:51.595837 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:51.595812 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-205.ec2.internal" podStartSLOduration=20.595800957 podStartE2EDuration="20.595800957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 20:00:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:00:35.550697057 +0000 UTC m=+5.598747403" watchObservedRunningTime="2026-09-11 20:00:51.595800957 +0000 UTC m=+21.643851301" Sep 11 20:00:51.610113 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:51.610070 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-2m6cs" podStartSLOduration=3.918083227 podStartE2EDuration="21.610055544s" totalImagesPullingTime="17.691972317s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 20:00:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:00:51.609502857 +0000 UTC m=+21.657553202" watchObservedRunningTime="2026-09-11 20:00:51.610055544 +0000 UTC m=+21.658105884" Sep 11 20:00:51.631202 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:51.630330 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-8kcqg" podStartSLOduration=3.869320922 podStartE2EDuration="21.630310835s" totalImagesPullingTime="17.760989913s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 20:00:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:00:51.629797546 +0000 UTC m=+21.677847892" watchObservedRunningTime="2026-09-11 20:00:51.630310835 +0000 UTC m=+21.678361162" Sep 11 20:00:51.668092 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:51.668047 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-jdhk4" podStartSLOduration=7.4968535769999995 podStartE2EDuration="21.668032382s" totalImagesPullingTime="14.171178805s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:00:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:00:51.667614046 +0000 UTC m=+21.715664389" watchObservedRunningTime="2026-09-11 20:00:51.668032382 +0000 UTC m=+21.716082725" Sep 11 20:00:51.781533 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:51.781503 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:51.858688 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:51.858507 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:51.959977 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:51.959951 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:52.021249 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:52.021218 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:52.183688 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:52.183589 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-jdhk4" Sep 11 20:00:52.184333 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:52.184312 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-jdhk4" Sep 11 20:00:52.444160 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:52.444088 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2dm2" Sep 11 20:00:52.444312 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:52.444088 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wk9hk" Sep 11 20:00:52.444312 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:52.444192 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2dm2" podUID="1f86594f-112f-4e58-a2ca-bbd951c4d1af" Sep 11 20:00:52.444312 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:52.444088 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zpl5q" Sep 11 20:00:52.444312 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:52.444253 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wk9hk" podUID="ae2b5d1f-4bb1-41eb-9096-d54b304ad672" Sep 11 20:00:52.444484 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:52.444365 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zpl5q" podUID="ce96d552-27b5-4912-86b0-38f0f46d0002" Sep 11 20:00:52.590249 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:52.590196 2546 generic.go:356] "Generic (PLEG): container finished" podID="52c72e1a-d061-4d3c-922b-b572a6496fed" containerID="ef5842bbc5b587b4f43d4c3d85073aba6c0f2840d0c3132b713a00d8c971b146" exitCode=0 Sep 11 20:00:52.591038 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:52.590275 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ddmz9" event={"ID":"52c72e1a-d061-4d3c-922b-b572a6496fed","Type":"ContainerDied","Data":"ef5842bbc5b587b4f43d4c3d85073aba6c0f2840d0c3132b713a00d8c971b146"} Sep 11 20:00:52.593897 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:52.593873 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" event={"ID":"b6b30385-b0e1-4ab5-916a-57992046f196","Type":"ContainerStarted","Data":"0f57bb037f88cc1f91cc009408facb23a0bb39504dd9bb7fd284325d399afce6"} Sep 11 20:00:52.593995 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:52.593955 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" event={"ID":"b6b30385-b0e1-4ab5-916a-57992046f196","Type":"ContainerStarted","Data":"aabe1525155ffb90a12595150bb3b36d8e9db4a278874fd10ef2ba35e126edfb"} Sep 11 20:00:52.594058 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:52.594008 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" event={"ID":"b6b30385-b0e1-4ab5-916a-57992046f196","Type":"ContainerStarted","Data":"9cb204e4e7151bd9736f05477bcb8f297c7eb4738ba986d8a52a4ba5c5d5b6a1"} Sep 11 20:00:52.594114 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:52.594058 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" event={"ID":"b6b30385-b0e1-4ab5-916a-57992046f196","Type":"ContainerStarted","Data":"4c2eb68bbf3b38d58cfc9320bedf04b54b09db57e5a499c7b6662fa41ea7c932"} Sep 11 20:00:52.594114 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:52.594106 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" event={"ID":"b6b30385-b0e1-4ab5-916a-57992046f196","Type":"ContainerStarted","Data":"1550508c531bf32eae0019ad425e0b2865174c1b8442515f2f46fbb7bf4093d7"} Sep 11 20:00:52.595623 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:52.595598 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-54fff" event={"ID":"6e8ee8b7-3b67-4b9d-88fc-552fd655d11a","Type":"ContainerStarted","Data":"e043f8fd61718a8e1e24cf414e7ea8f9b5db0a91cbc0e116fb1bf7a5a7b15e40"} Sep 11 20:00:52.597261 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:52.597230 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-r64rv" event={"ID":"482b4b76-dc42-490f-afcb-5b672806c329","Type":"ContainerStarted","Data":"437bb578e703d1b348520dc03228e08873d098e4558cadac04b0072f2ecc3c3f"} Sep 11 20:00:52.597963 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:52.597945 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-jdhk4" Sep 11 20:00:52.598397 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:52.598380 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jdhk4" Sep 11 20:00:52.625194 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:52.625152 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-r64rv" podStartSLOduration=4.932067661 podStartE2EDuration="22.625139059s" totalImagesPullingTime="17.693071398s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 20:00:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:00:52.624895906 +0000 UTC m=+22.672946250" watchObservedRunningTime="2026-09-11 20:00:52.625139059 +0000 UTC m=+22.673189400" Sep 11 20:00:52.660882 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:52.660833 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-54fff" podStartSLOduration=4.953354432 podStartE2EDuration="22.660817498s" totalImagesPullingTime="17.707463066s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 20:00:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:00:52.66052879 +0000 UTC m=+22.708579133" watchObservedRunningTime="2026-09-11 20:00:52.660817498 +0000 UTC m=+22.708867851" Sep 11 20:00:52.749410 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:52.749342 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:52.851830 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:52.851806 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:52.877695 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:52.877675 2546 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 11 20:00:53.102676 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:53.102592 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:53.203709 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:53.203680 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:53.424618 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:53.424407 2546 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T20:00:52.877689911Z","UUID":"9f411b7a-f2a3-4841-abed-e199c0dfff6a","Handler":null,"Name":"","Endpoint":""} Sep 11 20:00:53.426010 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:53.425990 2546 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 11 20:00:53.426111 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:53.426025 2546 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 11 20:00:53.601109 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:53.601065 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w6p5s" event={"ID":"52601b9c-beef-4b1c-aaf3-34cf267a00fe","Type":"ContainerStarted","Data":"0d28fe1e82bdeb158cbe01b59bfe9f48fe96ede811ec64ec339080f808265a4d"} Sep 11 20:00:53.902310 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:53.902211 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:54.307799 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:54.307748 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:54.407265 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:54.407227 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:54.447993 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:54.447916 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wk9hk" Sep 11 20:00:54.448151 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:54.448019 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2dm2" Sep 11 20:00:54.448151 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:54.448033 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wk9hk" podUID="ae2b5d1f-4bb1-41eb-9096-d54b304ad672" Sep 11 20:00:54.448151 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:54.448048 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zpl5q" Sep 11 20:00:54.448151 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:54.448129 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2dm2" podUID="1f86594f-112f-4e58-a2ca-bbd951c4d1af" Sep 11 20:00:54.448348 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:54.448206 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zpl5q" podUID="ce96d552-27b5-4912-86b0-38f0f46d0002" Sep 11 20:00:54.585504 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:54.585463 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:54.607550 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:54.607516 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" event={"ID":"b6b30385-b0e1-4ab5-916a-57992046f196","Type":"ContainerStarted","Data":"a8193d2342610eced36d1071556d73405626ae7df1bf0f387dbca8ab3ea2632b"} Sep 11 20:00:54.610297 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:54.610270 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w6p5s" event={"ID":"52601b9c-beef-4b1c-aaf3-34cf267a00fe","Type":"ContainerStarted","Data":"3c0537fd1d4b16f6719bad431f9749da5ff6412f420b2005993b4c5f1b3302e1"} Sep 11 20:00:54.627800 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:54.627747 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w6p5s" podStartSLOduration=4.387313921 podStartE2EDuration="24.627725607s" totalImagesPullingTime="20.240411686s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 20:00:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:00:54.627141189 +0000 UTC m=+24.675191533" watchObservedRunningTime="2026-09-11 20:00:54.627725607 +0000 UTC m=+24.675775966" Sep 11 20:00:54.685752 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:54.685719 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:56.444173 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:56.443987 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2dm2" Sep 11 20:00:56.444641 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:56.444002 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zpl5q" Sep 11 20:00:56.444641 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:56.444066 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wk9hk" Sep 11 20:00:56.444641 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:56.444271 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2dm2" podUID="1f86594f-112f-4e58-a2ca-bbd951c4d1af" Sep 11 20:00:56.444641 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:56.444351 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wk9hk" podUID="ae2b5d1f-4bb1-41eb-9096-d54b304ad672" Sep 11 20:00:56.444641 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:56.444459 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zpl5q" podUID="ce96d552-27b5-4912-86b0-38f0f46d0002" Sep 11 20:00:57.618849 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:57.618816 2546 generic.go:356] "Generic (PLEG): container finished" podID="52c72e1a-d061-4d3c-922b-b572a6496fed" containerID="2eea5e13a8bc9e91f135a9b4528faf047fa5fecf1944fe955b7b7e36e37a5756" exitCode=0 Sep 11 20:00:57.619771 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:57.618898 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ddmz9" event={"ID":"52c72e1a-d061-4d3c-922b-b572a6496fed","Type":"ContainerDied","Data":"2eea5e13a8bc9e91f135a9b4528faf047fa5fecf1944fe955b7b7e36e37a5756"} Sep 11 20:00:57.622700 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:57.622674 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" event={"ID":"b6b30385-b0e1-4ab5-916a-57992046f196","Type":"ContainerStarted","Data":"cfff751dbf6c2ef52a23bf909d81eabaa473049fcc9838b02abee82cc5e7b373"} Sep 11 20:00:57.622861 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:57.622840 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:57.622940 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:57.622918 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:57.622987 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:57.622957 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:57.638491 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:57.638469 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:57.638629 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:57.638561 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:00:57.666989 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:57.666932 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" podStartSLOduration=9.90572604 podStartE2EDuration="27.666917308s" totalImagesPullingTime="17.761191268s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 20:00:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:00:57.66614485 +0000 UTC m=+27.714195203" watchObservedRunningTime="2026-09-11 20:00:57.666917308 +0000 UTC m=+27.714967675" Sep 11 20:00:58.444497 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:58.444329 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2dm2" Sep 11 20:00:58.444681 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:58.444350 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zpl5q" Sep 11 20:00:58.444776 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:58.444374 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wk9hk" Sep 11 20:00:58.444776 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:58.444761 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zpl5q" podUID="ce96d552-27b5-4912-86b0-38f0f46d0002" Sep 11 20:00:58.444894 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:58.444621 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2dm2" podUID="1f86594f-112f-4e58-a2ca-bbd951c4d1af" Sep 11 20:00:58.444894 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:58.444857 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wk9hk" podUID="ae2b5d1f-4bb1-41eb-9096-d54b304ad672" Sep 11 20:00:58.841454 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:58.841414 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wk9hk"] Sep 11 20:00:58.842042 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:58.841534 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wk9hk" Sep 11 20:00:58.842042 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:58.841662 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wk9hk" podUID="ae2b5d1f-4bb1-41eb-9096-d54b304ad672" Sep 11 20:00:58.842750 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:58.842729 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zpl5q"] Sep 11 20:00:58.842844 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:58.842804 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zpl5q" Sep 11 20:00:58.842888 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:58.842875 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zpl5q" podUID="ce96d552-27b5-4912-86b0-38f0f46d0002" Sep 11 20:00:58.851891 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:58.851870 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-q2dm2"] Sep 11 20:00:58.851997 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:58.851947 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2dm2" Sep 11 20:00:58.852060 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:00:58.852044 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2dm2" podUID="1f86594f-112f-4e58-a2ca-bbd951c4d1af" Sep 11 20:00:59.006884 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:59.006817 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:59.108673 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:59.108647 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:59.176714 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:59.176688 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:59.629733 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:59.629703 2546 generic.go:356] "Generic (PLEG): container finished" podID="52c72e1a-d061-4d3c-922b-b572a6496fed" containerID="919ae33906bf90eac04abbcf55ed89e12a2b4860a132c985e45f24548b1b395b" exitCode=0 Sep 11 20:00:59.629920 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:59.629782 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ddmz9" event={"ID":"52c72e1a-d061-4d3c-922b-b572a6496fed","Type":"ContainerDied","Data":"919ae33906bf90eac04abbcf55ed89e12a2b4860a132c985e45f24548b1b395b"} Sep 11 20:00:59.819299 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:59.819263 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:59.892886 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:00:59.892793 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:00.364113 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:00.364085 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:00.444404 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:00.444379 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wk9hk" Sep 11 20:01:00.444519 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:00.444455 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2dm2" Sep 11 20:01:00.444519 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:01:00.444474 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wk9hk" podUID="ae2b5d1f-4bb1-41eb-9096-d54b304ad672" Sep 11 20:01:00.444628 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:01:00.444557 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2dm2" podUID="1f86594f-112f-4e58-a2ca-bbd951c4d1af" Sep 11 20:01:00.444628 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:00.444570 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zpl5q" Sep 11 20:01:00.444704 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:01:00.444683 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zpl5q" podUID="ce96d552-27b5-4912-86b0-38f0f46d0002" Sep 11 20:01:00.460219 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:00.460188 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:00.559896 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:00.559872 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:00.634722 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:00.634699 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ddmz9" event={"ID":"52c72e1a-d061-4d3c-922b-b572a6496fed","Type":"ContainerStarted","Data":"4bdf7021954c33052007372c7eb15277214bce4a01a0a3abc880f1dc87ac2e99"} Sep 11 20:01:00.778224 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:00.778188 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2m6cs_2133a2c0-f263-45aa-9d96-1819721a8c9b/dns-node-resolver/0.log" Sep 11 20:01:00.861998 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:00.861965 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:00.921441 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:00.921353 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:01.155391 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:01.155182 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:01.273930 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:01.273858 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:01.639381 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:01.639352 2546 generic.go:356] "Generic (PLEG): container finished" podID="52c72e1a-d061-4d3c-922b-b572a6496fed" containerID="4bdf7021954c33052007372c7eb15277214bce4a01a0a3abc880f1dc87ac2e99" exitCode=0 Sep 11 20:01:01.639536 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:01.639389 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ddmz9" event={"ID":"52c72e1a-d061-4d3c-922b-b572a6496fed","Type":"ContainerDied","Data":"4bdf7021954c33052007372c7eb15277214bce4a01a0a3abc880f1dc87ac2e99"} Sep 11 20:01:01.770079 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:01.770048 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:01.778569 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:01.778546 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-54fff_6e8ee8b7-3b67-4b9d-88fc-552fd655d11a/node-ca/0.log" Sep 11 20:01:01.836007 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:01.835979 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:01.891302 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:01.891233 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:02.214087 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:02.214001 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:02.313475 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:02.313447 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:02.443890 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:02.443857 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zpl5q" Sep 11 20:01:02.444082 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:02.443857 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2dm2" Sep 11 20:01:02.444082 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:01:02.443987 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zpl5q" podUID="ce96d552-27b5-4912-86b0-38f0f46d0002" Sep 11 20:01:02.444082 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:02.444044 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wk9hk" Sep 11 20:01:02.444310 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:01:02.444188 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2dm2" podUID="1f86594f-112f-4e58-a2ca-bbd951c4d1af" Sep 11 20:01:02.444310 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:01:02.444271 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wk9hk" podUID="ae2b5d1f-4bb1-41eb-9096-d54b304ad672" Sep 11 20:01:02.976851 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:02.976807 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:03.077938 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:03.077906 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:04.093487 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:04.093456 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:04.119618 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:04.119568 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1f86594f-112f-4e58-a2ca-bbd951c4d1af-original-pull-secret\") pod \"global-pull-secret-syncer-q2dm2\" (UID: \"1f86594f-112f-4e58-a2ca-bbd951c4d1af\") " pod="kube-system/global-pull-secret-syncer-q2dm2" Sep 11 20:01:04.119753 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:04.119626 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwhtf\" (UniqueName: \"kubernetes.io/projected/ae2b5d1f-4bb1-41eb-9096-d54b304ad672-kube-api-access-cwhtf\") pod \"network-check-target-wk9hk\" (UID: \"ae2b5d1f-4bb1-41eb-9096-d54b304ad672\") " pod="openshift-network-diagnostics/network-check-target-wk9hk" Sep 11 20:01:04.119753 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:01:04.119683 2546 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 20:01:04.119753 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:01:04.119729 2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 20:01:04.119753 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:01:04.119745 2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 20:01:04.119911 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:01:04.119759 2546 projected.go:196] Error preparing data for projected volume kube-api-access-cwhtf for pod openshift-network-diagnostics/network-check-target-wk9hk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 20:01:04.119911 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:01:04.119747 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f86594f-112f-4e58-a2ca-bbd951c4d1af-original-pull-secret podName:1f86594f-112f-4e58-a2ca-bbd951c4d1af nodeName:}" failed. No retries permitted until 2026-09-11 20:01:36.119729807 +0000 UTC m=+66.167780129 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1f86594f-112f-4e58-a2ca-bbd951c4d1af-original-pull-secret") pod "global-pull-secret-syncer-q2dm2" (UID: "1f86594f-112f-4e58-a2ca-bbd951c4d1af") : object "kube-system"/"original-pull-secret" not registered Sep 11 20:01:04.119911 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:01:04.119805 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae2b5d1f-4bb1-41eb-9096-d54b304ad672-kube-api-access-cwhtf podName:ae2b5d1f-4bb1-41eb-9096-d54b304ad672 nodeName:}" failed. No retries permitted until 2026-09-11 20:01:36.119792925 +0000 UTC m=+66.167843248 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-cwhtf" (UniqueName: "kubernetes.io/projected/ae2b5d1f-4bb1-41eb-9096-d54b304ad672-kube-api-access-cwhtf") pod "network-check-target-wk9hk" (UID: "ae2b5d1f-4bb1-41eb-9096-d54b304ad672") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 20:01:04.220730 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:04.220642 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce96d552-27b5-4912-86b0-38f0f46d0002-metrics-certs\") pod \"network-metrics-daemon-zpl5q\" (UID: \"ce96d552-27b5-4912-86b0-38f0f46d0002\") " pod="openshift-multus/network-metrics-daemon-zpl5q" Sep 11 20:01:04.220862 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:01:04.220809 2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 20:01:04.220897 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:01:04.220875 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce96d552-27b5-4912-86b0-38f0f46d0002-metrics-certs podName:ce96d552-27b5-4912-86b0-38f0f46d0002 nodeName:}" failed. No retries permitted until 2026-09-11 20:01:36.220855105 +0000 UTC m=+66.268905427 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ce96d552-27b5-4912-86b0-38f0f46d0002-metrics-certs") pod "network-metrics-daemon-zpl5q" (UID: "ce96d552-27b5-4912-86b0-38f0f46d0002") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 20:01:04.260955 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:04.260932 2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-205.ec2.internal" event="NodeReady" Sep 11 20:01:04.261115 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:04.261073 2546 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 11 20:01:04.444226 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:04.444191 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zpl5q" Sep 11 20:01:04.444426 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:04.444191 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2dm2" Sep 11 20:01:04.444426 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:04.444192 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wk9hk" Sep 11 20:01:04.447060 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:04.447034 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-b7zd7\"" Sep 11 20:01:04.448662 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:04.448640 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 11 20:01:04.448783 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:04.448644 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 11 20:01:04.448783 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:04.448747 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 11 20:01:04.448783 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:04.448747 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 11 20:01:04.449002 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:04.448985 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fv8gc\"" Sep 11 20:01:05.068867 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:05.068833 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:05.227231 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:05.227189 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:05.329341 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:05.329213 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:05.428314 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:05.428279 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:05.529592 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:05.529550 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:05.627759 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:05.627728 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:05.947523 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:05.947428 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:06.654594 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:06.654544 2546 generic.go:356] "Generic (PLEG): container finished" podID="52c72e1a-d061-4d3c-922b-b572a6496fed" containerID="c1b063ba235ecced6d96818a2fc4d430e520f7c9ae8351631e98b60aa6805095" exitCode=0 Sep 11 20:01:06.655156 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:06.654611 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ddmz9" event={"ID":"52c72e1a-d061-4d3c-922b-b572a6496fed","Type":"ContainerDied","Data":"c1b063ba235ecced6d96818a2fc4d430e520f7c9ae8351631e98b60aa6805095"} Sep 11 20:01:06.945900 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:06.945806 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:07.131998 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:07.131966 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:07.339470 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:07.339440 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:07.545962 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:07.545928 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:07.660079 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:07.659996 2546 generic.go:356] "Generic (PLEG): container finished" podID="52c72e1a-d061-4d3c-922b-b572a6496fed" containerID="78ff50f0c66fe201c39ce1cbea9fd7a6de3216aaec95ad29badf4cb338853b05" exitCode=0 Sep 11 20:01:07.660079 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:07.660042 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ddmz9" event={"ID":"52c72e1a-d061-4d3c-922b-b572a6496fed","Type":"ContainerDied","Data":"78ff50f0c66fe201c39ce1cbea9fd7a6de3216aaec95ad29badf4cb338853b05"} Sep 11 20:01:07.737418 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:07.737386 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:08.141013 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:08.140981 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:08.565115 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:08.565038 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:08.629716 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:08.629687 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:08.666127 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:08.666095 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ddmz9" event={"ID":"52c72e1a-d061-4d3c-922b-b572a6496fed","Type":"ContainerStarted","Data":"051367c37516b88c5d1cef1e6e95ca688703492d04722517caeb1aae884aa71c"} Sep 11 20:01:08.687014 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:08.686963 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-ddmz9" podStartSLOduration=10.838515815000001 podStartE2EDuration="38.68694691s" totalImagesPullingTime="27.848431095s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 20:00:30 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:01:08.68583718 +0000 UTC m=+38.733887523" watchObservedRunningTime="2026-09-11 20:01:08.68694691 +0000 UTC m=+38.734997254" Sep 11 20:01:08.949706 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:08.949671 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:09.148445 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:09.148414 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:09.251408 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:09.251325 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:09.560183 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:09.560102 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:09.757353 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:09.757317 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:09.833402 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:09.833309 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:09.937696 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:09.937664 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:10.038676 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:10.038649 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:10.143343 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:10.143311 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:10.245791 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:10.245763 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:10.394305 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:10.394231 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:10.740687 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:10.740607 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:10.840951 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:10.840915 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:10.993290 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:10.993212 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:11.101012 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:11.098990 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:11.194753 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:11.194718 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:11.333679 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:11.333603 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:11.561536 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:11.561506 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:11.738051 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:11.738022 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:11.838864 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:11.838832 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:11.938989 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:11.938955 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:12.138005 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:12.137962 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:12.245352 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:12.245324 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:12.348200 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:12.348165 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:12.738639 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:12.738601 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:13.135377 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:13.135349 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:13.542374 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:13.542293 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:13.732899 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:13.732858 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:14.065736 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:14.065703 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:14.168749 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:14.168717 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:14.275548 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:14.275519 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:14.938050 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:14.938014 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:15.347143 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:15.347109 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:15.501246 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:15.501211 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:15.560607 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:15.560552 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:15.736459 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:15.736377 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:15.941795 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:15.941764 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:16.337880 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:16.337842 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:18.631493 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:18.631461 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:18.684256 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:18.684227 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:23.090021 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:23.089982 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:23.196282 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:23.196248 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:23.198591 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:23.198548 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rvgwn"] Sep 11 20:01:23.200173 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:23.200154 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rvgwn" Sep 11 20:01:23.202991 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:23.202967 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-82bcx"] Sep 11 20:01:23.203420 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:23.203393 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nbwp6\"" Sep 11 20:01:23.203610 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:23.203590 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 11 20:01:23.203757 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:23.203733 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 11 20:01:23.204600 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:23.204567 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-82bcx" Sep 11 20:01:23.206812 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:23.206796 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 11 20:01:23.206920 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:23.206835 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 11 20:01:23.207039 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:23.207017 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-fjh5z\"" Sep 11 20:01:23.207669 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:23.207311 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 11 20:01:23.213652 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:23.213511 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rvgwn"] Sep 11 20:01:23.216524 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:23.216504 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-82bcx"] Sep 11 20:01:23.222249 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:23.222229 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-gxt9c"] Sep 11 20:01:23.223964 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:23.223948 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gxt9c" Sep 11 20:01:23.226168 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:23.226151 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zqht8\"" Sep 11 20:01:23.226250 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:23.226185 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 11 20:01:23.226250 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:23.226151 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 11 20:01:23.226497 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:23.226483 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 11 20:01:23.226544 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:23.226498 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 11 20:01:23.237640 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:23.237620 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gxt9c"] Sep 11 20:01:23.302927 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:23.302892 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:23.357064 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:23.356981 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/488ce64f-e146-4301-8f4f-a8ca95abf4ae-config-volume\") pod \"dns-default-rvgwn\" (UID: \"488ce64f-e146-4301-8f4f-a8ca95abf4ae\") " pod="openshift-dns/dns-default-rvgwn" Sep 11 20:01:23.357064 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:23.357025 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5j5h\" (UniqueName: \"kubernetes.io/projected/1fdd17f8-61f0-45b2-a8d9-809c6d83ff03-kube-api-access-f5j5h\") pod \"ingress-canary-82bcx\" (UID: \"1fdd17f8-61f0-45b2-a8d9-809c6d83ff03\") " pod="openshift-ingress-canary/ingress-canary-82bcx" Sep 11 20:01:23.357224 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:23.357067 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9ed8a847-72ef-440f-a4f2-dae8cdb83925-data-volume\") pod \"insights-runtime-extractor-gxt9c\" (UID: \"9ed8a847-72ef-440f-a4f2-dae8cdb83925\") " pod="openshift-insights/insights-runtime-extractor-gxt9c" Sep 11 20:01:23.357224 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:23.357090 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trggr\" (UniqueName: \"kubernetes.io/projected/488ce64f-e146-4301-8f4f-a8ca95abf4ae-kube-api-access-trggr\") pod \"dns-default-rvgwn\" (UID: \"488ce64f-e146-4301-8f4f-a8ca95abf4ae\") " pod="openshift-dns/dns-default-rvgwn" Sep 11 20:01:23.357224 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:23.357173 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9ed8a847-72ef-440f-a4f2-dae8cdb83925-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gxt9c\" (UID: \"9ed8a847-72ef-440f-a4f2-dae8cdb83925\") " pod="openshift-insights/insights-runtime-extractor-gxt9c" Sep 11 20:01:23.357366 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:23.357228 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1fdd17f8-61f0-45b2-a8d9-809c6d83ff03-cert\") pod \"ingress-canary-82bcx\" (UID: \"1fdd17f8-61f0-45b2-a8d9-809c6d83ff03\") " pod="openshift-ingress-canary/ingress-canary-82bcx" Sep 11 20:01:23.357366 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:23.357262 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9ed8a847-72ef-440f-a4f2-dae8cdb83925-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gxt9c\" (UID: \"9ed8a847-72ef-440f-a4f2-dae8cdb83925\") " pod="openshift-insights/insights-runtime-extractor-gxt9c" Sep 11 20:01:23.357366 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:23.357303 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/488ce64f-e146-4301-8f4f-a8ca95abf4ae-tmp-dir\") pod \"dns-default-rvgwn\" (UID: \"488ce64f-e146-4301-8f4f-a8ca95abf4ae\") " pod="openshift-dns/dns-default-rvgwn" Sep 11 20:01:23.357366 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:23.357343 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j255p\" (UniqueName: \"kubernetes.io/projected/9ed8a847-72ef-440f-a4f2-dae8cdb83925-kube-api-access-j255p\") pod \"insights-runtime-extractor-gxt9c\" (UID: \"9ed8a847-72ef-440f-a4f2-dae8cdb83925\") " pod="openshift-insights/insights-runtime-extractor-gxt9c" Sep 11 20:01:23.357501 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:23.357400 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9ed8a847-72ef-440f-a4f2-dae8cdb83925-crio-socket\") pod \"insights-runtime-extractor-gxt9c\" (UID: \"9ed8a847-72ef-440f-a4f2-dae8cdb83925\") " pod="openshift-insights/insights-runtime-extractor-gxt9c" Sep 11 20:01:23.357501 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:23.357425 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/488ce64f-e146-4301-8f4f-a8ca95abf4ae-metrics-tls\") pod \"dns-default-rvgwn\" (UID: \"488ce64f-e146-4301-8f4f-a8ca95abf4ae\") " pod="openshift-dns/dns-default-rvgwn" Sep 11 20:01:23.451081 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:23.451050 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:23.458347 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:23.458313 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9ed8a847-72ef-440f-a4f2-dae8cdb83925-data-volume\") pod \"insights-runtime-extractor-gxt9c\" (UID: \"9ed8a847-72ef-440f-a4f2-dae8cdb83925\") " pod="openshift-insights/insights-runtime-extractor-gxt9c" Sep 11 20:01:23.458347 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:23.458353 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trggr\" (UniqueName: \"kubernetes.io/projected/488ce64f-e146-4301-8f4f-a8ca95abf4ae-kube-api-access-trggr\") pod \"dns-default-rvgwn\" (UID: \"488ce64f-e146-4301-8f4f-a8ca95abf4ae\") " pod="openshift-dns/dns-default-rvgwn" Sep 11 20:01:23.458466 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:23.458372 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9ed8a847-72ef-440f-a4f2-dae8cdb83925-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gxt9c\" (UID: \"9ed8a847-72ef-440f-a4f2-dae8cdb83925\") " pod="openshift-insights/insights-runtime-extractor-gxt9c" Sep 11 20:01:23.458466 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:23.458407 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1fdd17f8-61f0-45b2-a8d9-809c6d83ff03-cert\") pod \"ingress-canary-82bcx\" (UID: \"1fdd17f8-61f0-45b2-a8d9-809c6d83ff03\") " pod="openshift-ingress-canary/ingress-canary-82bcx" Sep 11 20:01:23.458466 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:23.458434 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9ed8a847-72ef-440f-a4f2-dae8cdb83925-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gxt9c\" (UID: \"9ed8a847-72ef-440f-a4f2-dae8cdb83925\") " pod="openshift-insights/insights-runtime-extractor-gxt9c" Sep 11 20:01:23.458466 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:23.458452 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/488ce64f-e146-4301-8f4f-a8ca95abf4ae-tmp-dir\") pod \"dns-default-rvgwn\" (UID: \"488ce64f-e146-4301-8f4f-a8ca95abf4ae\") " pod="openshift-dns/dns-default-rvgwn" Sep 11 20:01:23.458686 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:23.458469 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j255p\" (UniqueName: \"kubernetes.io/projected/9ed8a847-72ef-440f-a4f2-dae8cdb83925-kube-api-access-j255p\") pod \"insights-runtime-extractor-gxt9c\" (UID: \"9ed8a847-72ef-440f-a4f2-dae8cdb83925\") " pod="openshift-insights/insights-runtime-extractor-gxt9c" Sep 11 20:01:23.458686 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:23.458489 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9ed8a847-72ef-440f-a4f2-dae8cdb83925-crio-socket\") pod \"insights-runtime-extractor-gxt9c\" (UID: \"9ed8a847-72ef-440f-a4f2-dae8cdb83925\") " pod="openshift-insights/insights-runtime-extractor-gxt9c" Sep 11 20:01:23.458686 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:23.458505 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/488ce64f-e146-4301-8f4f-a8ca95abf4ae-metrics-tls\") pod \"dns-default-rvgwn\" (UID: \"488ce64f-e146-4301-8f4f-a8ca95abf4ae\") " pod="openshift-dns/dns-default-rvgwn" Sep 11 20:01:23.458825 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:23.458718 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9ed8a847-72ef-440f-a4f2-dae8cdb83925-data-volume\") pod \"insights-runtime-extractor-gxt9c\" (UID: \"9ed8a847-72ef-440f-a4f2-dae8cdb83925\") " pod="openshift-insights/insights-runtime-extractor-gxt9c" Sep 11 20:01:23.458871 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:23.458854 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/488ce64f-e146-4301-8f4f-a8ca95abf4ae-tmp-dir\") pod \"dns-default-rvgwn\" (UID: \"488ce64f-e146-4301-8f4f-a8ca95abf4ae\") " pod="openshift-dns/dns-default-rvgwn" Sep 11 20:01:23.458915 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:23.458855 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/488ce64f-e146-4301-8f4f-a8ca95abf4ae-config-volume\") pod \"dns-default-rvgwn\" (UID: \"488ce64f-e146-4301-8f4f-a8ca95abf4ae\") " pod="openshift-dns/dns-default-rvgwn" Sep 11 20:01:23.458915 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:23.458901 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9ed8a847-72ef-440f-a4f2-dae8cdb83925-crio-socket\") pod \"insights-runtime-extractor-gxt9c\" (UID: \"9ed8a847-72ef-440f-a4f2-dae8cdb83925\") " pod="openshift-insights/insights-runtime-extractor-gxt9c" Sep 11 20:01:23.458915 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:23.458901 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5j5h\" (UniqueName: \"kubernetes.io/projected/1fdd17f8-61f0-45b2-a8d9-809c6d83ff03-kube-api-access-f5j5h\") pod \"ingress-canary-82bcx\" (UID: \"1fdd17f8-61f0-45b2-a8d9-809c6d83ff03\") " pod="openshift-ingress-canary/ingress-canary-82bcx" Sep 11 20:01:23.459200 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:23.459175 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9ed8a847-72ef-440f-a4f2-dae8cdb83925-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gxt9c\" (UID: \"9ed8a847-72ef-440f-a4f2-dae8cdb83925\") " pod="openshift-insights/insights-runtime-extractor-gxt9c" Sep 11 20:01:23.459342 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:23.459321 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/488ce64f-e146-4301-8f4f-a8ca95abf4ae-config-volume\") pod \"dns-default-rvgwn\" (UID: \"488ce64f-e146-4301-8f4f-a8ca95abf4ae\") " pod="openshift-dns/dns-default-rvgwn" Sep 11 20:01:23.462110 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:23.462087 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/488ce64f-e146-4301-8f4f-a8ca95abf4ae-metrics-tls\") pod \"dns-default-rvgwn\" (UID: \"488ce64f-e146-4301-8f4f-a8ca95abf4ae\") " pod="openshift-dns/dns-default-rvgwn" Sep 11 20:01:23.462183 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:23.462127 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9ed8a847-72ef-440f-a4f2-dae8cdb83925-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gxt9c\" (UID: \"9ed8a847-72ef-440f-a4f2-dae8cdb83925\") " pod="openshift-insights/insights-runtime-extractor-gxt9c" Sep 11 20:01:23.462282 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:23.462267 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1fdd17f8-61f0-45b2-a8d9-809c6d83ff03-cert\") pod \"ingress-canary-82bcx\" (UID: \"1fdd17f8-61f0-45b2-a8d9-809c6d83ff03\") " pod="openshift-ingress-canary/ingress-canary-82bcx" Sep 11 20:01:23.466305 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:23.466280 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trggr\" (UniqueName: \"kubernetes.io/projected/488ce64f-e146-4301-8f4f-a8ca95abf4ae-kube-api-access-trggr\") pod \"dns-default-rvgwn\" (UID: \"488ce64f-e146-4301-8f4f-a8ca95abf4ae\") " pod="openshift-dns/dns-default-rvgwn" Sep 11 20:01:23.467540 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:23.467524 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j255p\" (UniqueName: \"kubernetes.io/projected/9ed8a847-72ef-440f-a4f2-dae8cdb83925-kube-api-access-j255p\") pod \"insights-runtime-extractor-gxt9c\" (UID: \"9ed8a847-72ef-440f-a4f2-dae8cdb83925\") " pod="openshift-insights/insights-runtime-extractor-gxt9c" Sep 11 20:01:23.471233 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:23.471214 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5j5h\" (UniqueName: \"kubernetes.io/projected/1fdd17f8-61f0-45b2-a8d9-809c6d83ff03-kube-api-access-f5j5h\") pod \"ingress-canary-82bcx\" (UID: \"1fdd17f8-61f0-45b2-a8d9-809c6d83ff03\") " pod="openshift-ingress-canary/ingress-canary-82bcx" Sep 11 20:01:23.511008 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:23.510975 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rvgwn" Sep 11 20:01:23.515567 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:23.515540 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-82bcx" Sep 11 20:01:23.531663 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:23.531640 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gxt9c" Sep 11 20:01:23.793181 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:23.793147 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rvgwn"] Sep 11 20:01:23.796667 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:23.796639 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-82bcx"] Sep 11 20:01:23.799382 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:01:23.799231 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod488ce64f_e146_4301_8f4f_a8ca95abf4ae.slice/crio-d2169d668db113a073d8e9b88a634c6b7b55d5c02ac558e3c90b4bb50b4e3ba8 WatchSource:0}: Error finding container d2169d668db113a073d8e9b88a634c6b7b55d5c02ac558e3c90b4bb50b4e3ba8: Status 404 returned error can't find the container with id d2169d668db113a073d8e9b88a634c6b7b55d5c02ac558e3c90b4bb50b4e3ba8 Sep 11 20:01:23.800623 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:01:23.800598 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fdd17f8_61f0_45b2_a8d9_809c6d83ff03.slice/crio-5b400c99cf51f4ee83f306d7e3e4a0eeb5a9ac4108271c97f6410cdcea1d4313 WatchSource:0}: Error finding container 5b400c99cf51f4ee83f306d7e3e4a0eeb5a9ac4108271c97f6410cdcea1d4313: Status 404 returned error can't find the container with id 5b400c99cf51f4ee83f306d7e3e4a0eeb5a9ac4108271c97f6410cdcea1d4313 Sep 11 20:01:23.828677 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:23.828647 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gxt9c"] Sep 11 20:01:23.831927 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:01:23.831904 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ed8a847_72ef_440f_a4f2_dae8cdb83925.slice/crio-d19c139767993b02bbf7f45b8a35ac7e0dd464dfb2bd80a614d114692d10f8dd WatchSource:0}: Error finding container d19c139767993b02bbf7f45b8a35ac7e0dd464dfb2bd80a614d114692d10f8dd: Status 404 returned error can't find the container with id d19c139767993b02bbf7f45b8a35ac7e0dd464dfb2bd80a614d114692d10f8dd Sep 11 20:01:24.704779 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:24.704728 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-82bcx" event={"ID":"1fdd17f8-61f0-45b2-a8d9-809c6d83ff03","Type":"ContainerStarted","Data":"5b400c99cf51f4ee83f306d7e3e4a0eeb5a9ac4108271c97f6410cdcea1d4313"} Sep 11 20:01:24.706146 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:24.706115 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rvgwn" event={"ID":"488ce64f-e146-4301-8f4f-a8ca95abf4ae","Type":"ContainerStarted","Data":"d2169d668db113a073d8e9b88a634c6b7b55d5c02ac558e3c90b4bb50b4e3ba8"} Sep 11 20:01:24.708076 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:24.708054 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gxt9c" event={"ID":"9ed8a847-72ef-440f-a4f2-dae8cdb83925","Type":"ContainerStarted","Data":"6f612a8df402e40f031cf282074458f536692ef37833f12f84ccb1932660ed13"} Sep 11 20:01:24.708231 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:24.708102 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gxt9c" event={"ID":"9ed8a847-72ef-440f-a4f2-dae8cdb83925","Type":"ContainerStarted","Data":"aec1bce1fbeca02f7d4edddcb7914c39b061173f4ea37af790aaed9b9c54b12c"} Sep 11 20:01:24.708231 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:24.708130 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gxt9c" event={"ID":"9ed8a847-72ef-440f-a4f2-dae8cdb83925","Type":"ContainerStarted","Data":"d19c139767993b02bbf7f45b8a35ac7e0dd464dfb2bd80a614d114692d10f8dd"} Sep 11 20:01:25.714542 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:25.714446 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rvgwn" event={"ID":"488ce64f-e146-4301-8f4f-a8ca95abf4ae","Type":"ContainerStarted","Data":"d71ac0670e35b9c073fbeea460ee30faf996dcd76c3e39c8a4b40a3db69975ed"} Sep 11 20:01:25.714542 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:25.714519 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rvgwn" event={"ID":"488ce64f-e146-4301-8f4f-a8ca95abf4ae","Type":"ContainerStarted","Data":"3a9ad01ce844b7248489af0de4ca6ef8fef00ee6941bca684572c5f94fe966df"} Sep 11 20:01:25.715107 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:25.714739 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rvgwn" Sep 11 20:01:25.735216 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:25.735165 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-rvgwn" podStartSLOduration=1.121367736 podStartE2EDuration="2.735146812s" totalImagesPullingTime="1.613779076s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:01:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:01:25.734429079 +0000 UTC m=+55.782479425" watchObservedRunningTime="2026-09-11 20:01:25.735146812 +0000 UTC m=+55.783197157" Sep 11 20:01:27.240388 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:27.240349 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:27.414268 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:27.414234 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-8g4nj"] Sep 11 20:01:27.417106 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:27.417090 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-8g4nj" Sep 11 20:01:27.419452 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:27.419430 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 11 20:01:27.419495 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:27.419440 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-llkc9\"" Sep 11 20:01:27.422504 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:27.422483 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:27.438639 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:27.438615 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-8g4nj"] Sep 11 20:01:27.485919 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:27.485876 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a3200c2f-fb6b-4a21-83b4-3c0e04f084ae-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-8g4nj\" (UID: \"a3200c2f-fb6b-4a21-83b4-3c0e04f084ae\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-8g4nj" Sep 11 20:01:27.586759 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:27.586662 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a3200c2f-fb6b-4a21-83b4-3c0e04f084ae-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-8g4nj\" (UID: \"a3200c2f-fb6b-4a21-83b4-3c0e04f084ae\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-8g4nj" Sep 11 20:01:27.586908 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:01:27.586810 2546 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Sep 11 20:01:27.586908 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:01:27.586878 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3200c2f-fb6b-4a21-83b4-3c0e04f084ae-tls-certificates podName:a3200c2f-fb6b-4a21-83b4-3c0e04f084ae nodeName:}" failed. No retries permitted until 2026-09-11 20:01:28.086862663 +0000 UTC m=+58.134912998 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/a3200c2f-fb6b-4a21-83b4-3c0e04f084ae-tls-certificates") pod "prometheus-operator-admission-webhook-5864cd99d7-8g4nj" (UID: "a3200c2f-fb6b-4a21-83b4-3c0e04f084ae") : secret "prometheus-operator-admission-webhook-tls" not found Sep 11 20:01:27.722087 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:27.722049 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gxt9c" event={"ID":"9ed8a847-72ef-440f-a4f2-dae8cdb83925","Type":"ContainerStarted","Data":"276757ccf0604ed7ba38b847ad9af072d9c5d2fe792d8aaad341ba6ebbf119b5"} Sep 11 20:01:27.723541 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:27.723500 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-82bcx" event={"ID":"1fdd17f8-61f0-45b2-a8d9-809c6d83ff03","Type":"ContainerStarted","Data":"e0dc54896235bde5787c6646538f17203311b0292d1f042b11a020e164b031df"} Sep 11 20:01:27.743615 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:27.741925 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-gxt9c" podStartSLOduration=1.989943352 podStartE2EDuration="4.741905071s" totalImagesPullingTime="2.751961719s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 20:01:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:01:27.741600481 +0000 UTC m=+57.789650826" watchObservedRunningTime="2026-09-11 20:01:27.741905071 +0000 UTC m=+57.789955416" Sep 11 20:01:27.756887 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:27.756843 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-82bcx" podStartSLOduration=1.8297366290000001 podStartE2EDuration="4.756826249s" totalImagesPullingTime="2.92708962s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:01:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:01:27.756696923 +0000 UTC m=+57.804747268" watchObservedRunningTime="2026-09-11 20:01:27.756826249 +0000 UTC m=+57.804876594" Sep 11 20:01:28.064122 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:28.064084 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:28.090334 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:28.090294 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a3200c2f-fb6b-4a21-83b4-3c0e04f084ae-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-8g4nj\" (UID: \"a3200c2f-fb6b-4a21-83b4-3c0e04f084ae\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-8g4nj" Sep 11 20:01:28.092633 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:28.092601 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a3200c2f-fb6b-4a21-83b4-3c0e04f084ae-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-8g4nj\" (UID: \"a3200c2f-fb6b-4a21-83b4-3c0e04f084ae\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-8g4nj" Sep 11 20:01:28.124437 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:28.124403 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:28.326193 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:28.326096 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-8g4nj" Sep 11 20:01:28.592564 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:28.592536 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-8g4nj"] Sep 11 20:01:28.595759 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:01:28.595728 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3200c2f_fb6b_4a21_83b4_3c0e04f084ae.slice/crio-7250867e55f2fd1a7201b37c7384739084dcad9e99a74fdf5dd129c8c797b86c WatchSource:0}: Error finding container 7250867e55f2fd1a7201b37c7384739084dcad9e99a74fdf5dd129c8c797b86c: Status 404 returned error can't find the container with id 7250867e55f2fd1a7201b37c7384739084dcad9e99a74fdf5dd129c8c797b86c Sep 11 20:01:28.647738 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:28.647705 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5f4bb78b9b-gwm4b"] Sep 11 20:01:28.652720 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:28.652701 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f4bb78b9b-gwm4b" Sep 11 20:01:28.654755 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:28.654729 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-g7fth\"" Sep 11 20:01:28.654864 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:28.654805 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 11 20:01:28.654924 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:28.654880 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 11 20:01:28.655451 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:28.655435 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 11 20:01:28.655451 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:28.655452 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 11 20:01:28.655626 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:28.655523 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 11 20:01:28.655626 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:28.655603 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 11 20:01:28.655733 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:28.655721 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 11 20:01:28.663782 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:28.663763 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f4bb78b9b-gwm4b"] Sep 11 20:01:28.695615 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:28.695561 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6-service-ca\") pod \"console-5f4bb78b9b-gwm4b\" (UID: \"fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6\") " pod="openshift-console/console-5f4bb78b9b-gwm4b" Sep 11 20:01:28.695790 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:28.695626 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6-console-config\") pod \"console-5f4bb78b9b-gwm4b\" (UID: \"fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6\") " pod="openshift-console/console-5f4bb78b9b-gwm4b" Sep 11 20:01:28.695790 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:28.695710 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6-console-oauth-config\") pod \"console-5f4bb78b9b-gwm4b\" (UID: \"fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6\") " pod="openshift-console/console-5f4bb78b9b-gwm4b" Sep 11 20:01:28.695790 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:28.695752 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6-oauth-serving-cert\") pod \"console-5f4bb78b9b-gwm4b\" (UID: \"fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6\") " pod="openshift-console/console-5f4bb78b9b-gwm4b" Sep 11 20:01:28.695914 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:28.695791 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6-console-serving-cert\") pod \"console-5f4bb78b9b-gwm4b\" (UID: \"fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6\") " pod="openshift-console/console-5f4bb78b9b-gwm4b" Sep 11 20:01:28.695914 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:28.695824 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6mbs\" (UniqueName: \"kubernetes.io/projected/fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6-kube-api-access-b6mbs\") pod \"console-5f4bb78b9b-gwm4b\" (UID: \"fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6\") " pod="openshift-console/console-5f4bb78b9b-gwm4b" Sep 11 20:01:28.727328 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:28.727287 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-8g4nj" event={"ID":"a3200c2f-fb6b-4a21-83b4-3c0e04f084ae","Type":"ContainerStarted","Data":"7250867e55f2fd1a7201b37c7384739084dcad9e99a74fdf5dd129c8c797b86c"} Sep 11 20:01:28.796570 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:28.796532 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6-oauth-serving-cert\") pod \"console-5f4bb78b9b-gwm4b\" (UID: \"fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6\") " pod="openshift-console/console-5f4bb78b9b-gwm4b" Sep 11 20:01:28.796570 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:28.796592 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6-console-serving-cert\") pod \"console-5f4bb78b9b-gwm4b\" (UID: \"fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6\") " pod="openshift-console/console-5f4bb78b9b-gwm4b" Sep 11 20:01:28.796838 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:28.796612 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6mbs\" (UniqueName: \"kubernetes.io/projected/fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6-kube-api-access-b6mbs\") pod \"console-5f4bb78b9b-gwm4b\" (UID: \"fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6\") " pod="openshift-console/console-5f4bb78b9b-gwm4b" Sep 11 20:01:28.796838 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:28.796677 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6-service-ca\") pod \"console-5f4bb78b9b-gwm4b\" (UID: \"fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6\") " pod="openshift-console/console-5f4bb78b9b-gwm4b" Sep 11 20:01:28.796838 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:28.796747 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6-console-config\") pod \"console-5f4bb78b9b-gwm4b\" (UID: \"fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6\") " pod="openshift-console/console-5f4bb78b9b-gwm4b" Sep 11 20:01:28.796838 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:28.796788 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6-console-oauth-config\") pod \"console-5f4bb78b9b-gwm4b\" (UID: \"fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6\") " pod="openshift-console/console-5f4bb78b9b-gwm4b" Sep 11 20:01:28.797433 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:28.797405 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6-oauth-serving-cert\") pod \"console-5f4bb78b9b-gwm4b\" (UID: \"fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6\") " pod="openshift-console/console-5f4bb78b9b-gwm4b" Sep 11 20:01:28.797544 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:28.797520 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6-service-ca\") pod \"console-5f4bb78b9b-gwm4b\" (UID: \"fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6\") " pod="openshift-console/console-5f4bb78b9b-gwm4b" Sep 11 20:01:28.797613 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:28.797536 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6-console-config\") pod \"console-5f4bb78b9b-gwm4b\" (UID: \"fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6\") " pod="openshift-console/console-5f4bb78b9b-gwm4b" Sep 11 20:01:28.799149 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:28.799128 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6-console-oauth-config\") pod \"console-5f4bb78b9b-gwm4b\" (UID: \"fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6\") " pod="openshift-console/console-5f4bb78b9b-gwm4b" Sep 11 20:01:28.799234 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:28.799193 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6-console-serving-cert\") pod \"console-5f4bb78b9b-gwm4b\" (UID: \"fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6\") " pod="openshift-console/console-5f4bb78b9b-gwm4b" Sep 11 20:01:28.804641 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:28.804621 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6mbs\" (UniqueName: \"kubernetes.io/projected/fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6-kube-api-access-b6mbs\") pod \"console-5f4bb78b9b-gwm4b\" (UID: \"fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6\") " pod="openshift-console/console-5f4bb78b9b-gwm4b" Sep 11 20:01:28.961949 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:28.961871 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f4bb78b9b-gwm4b" Sep 11 20:01:29.088824 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:29.088790 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:29.173480 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:29.173443 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:29.252115 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:29.252048 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f4bb78b9b-gwm4b"] Sep 11 20:01:29.254914 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:01:29.254893 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbc2ea33_f0f9_4b0c_b1ae_b36f6cdae4e6.slice/crio-a4a516e1626f9806651a3bfe190f33195e609043939775df6390b2e21515af9b WatchSource:0}: Error finding container a4a516e1626f9806651a3bfe190f33195e609043939775df6390b2e21515af9b: Status 404 returned error can't find the container with id a4a516e1626f9806651a3bfe190f33195e609043939775df6390b2e21515af9b Sep 11 20:01:29.642403 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:29.642375 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pgnnf" Sep 11 20:01:29.734805 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:29.734773 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f4bb78b9b-gwm4b" event={"ID":"fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6","Type":"ContainerStarted","Data":"a4a516e1626f9806651a3bfe190f33195e609043939775df6390b2e21515af9b"} Sep 11 20:01:29.798395 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:29.798359 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:29.897985 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:29.897911 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:29.953960 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:29.953931 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:30.078594 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:30.078551 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:30.135443 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:30.135408 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:30.231526 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:30.231499 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:30.335556 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:30.335504 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:30.740286 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:30.740240 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-8g4nj" event={"ID":"a3200c2f-fb6b-4a21-83b4-3c0e04f084ae","Type":"ContainerStarted","Data":"92f76be9be8c068dd8b4a138ff9458beab0dc3dd061f110160491e213b7503d7"} Sep 11 20:01:30.740736 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:30.740463 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-8g4nj" Sep 11 20:01:30.745999 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:30.745973 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-8g4nj" Sep 11 20:01:30.757205 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:30.757150 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-8g4nj" podStartSLOduration=2.134939136 podStartE2EDuration="3.757130717s" totalImagesPullingTime="1.622191581s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:01:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:01:30.756296245 +0000 UTC m=+60.804346590" watchObservedRunningTime="2026-09-11 20:01:30.757130717 +0000 UTC m=+60.805181063" Sep 11 20:01:31.448048 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:31.448014 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:31.472146 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:31.472113 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-xrnbx"] Sep 11 20:01:31.475047 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:31.475015 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-xrnbx" Sep 11 20:01:31.477304 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:31.477242 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 11 20:01:31.477304 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:31.477290 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 11 20:01:31.477472 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:31.477322 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 11 20:01:31.477790 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:31.477740 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 11 20:01:31.477790 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:31.477783 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-ncwc6\"" Sep 11 20:01:31.477999 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:31.477978 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 11 20:01:31.485685 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:31.485658 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-xrnbx"] Sep 11 20:01:31.516518 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:31.516483 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:31.516709 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:31.516567 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e40d01fe-6f2e-4dce-95af-72a07ce53d56-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-xrnbx\" (UID: \"e40d01fe-6f2e-4dce-95af-72a07ce53d56\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-xrnbx" Sep 11 20:01:31.516709 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:31.516612 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc549\" (UniqueName: \"kubernetes.io/projected/e40d01fe-6f2e-4dce-95af-72a07ce53d56-kube-api-access-rc549\") pod \"prometheus-operator-68bcd7f987-xrnbx\" (UID: \"e40d01fe-6f2e-4dce-95af-72a07ce53d56\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-xrnbx" Sep 11 20:01:31.516709 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:31.516679 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e40d01fe-6f2e-4dce-95af-72a07ce53d56-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-xrnbx\" (UID: \"e40d01fe-6f2e-4dce-95af-72a07ce53d56\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-xrnbx" Sep 11 20:01:31.516849 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:31.516718 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e40d01fe-6f2e-4dce-95af-72a07ce53d56-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-xrnbx\" (UID: \"e40d01fe-6f2e-4dce-95af-72a07ce53d56\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-xrnbx" Sep 11 20:01:31.617506 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:31.617468 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e40d01fe-6f2e-4dce-95af-72a07ce53d56-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-xrnbx\" (UID: \"e40d01fe-6f2e-4dce-95af-72a07ce53d56\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-xrnbx" Sep 11 20:01:31.617698 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:31.617533 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e40d01fe-6f2e-4dce-95af-72a07ce53d56-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-xrnbx\" (UID: \"e40d01fe-6f2e-4dce-95af-72a07ce53d56\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-xrnbx" Sep 11 20:01:31.622176 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:31.617653 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e40d01fe-6f2e-4dce-95af-72a07ce53d56-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-xrnbx\" (UID: \"e40d01fe-6f2e-4dce-95af-72a07ce53d56\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-xrnbx" Sep 11 20:01:31.622176 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:01:31.618283 2546 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Sep 11 20:01:31.622176 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:31.618319 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc549\" (UniqueName: \"kubernetes.io/projected/e40d01fe-6f2e-4dce-95af-72a07ce53d56-kube-api-access-rc549\") pod \"prometheus-operator-68bcd7f987-xrnbx\" (UID: \"e40d01fe-6f2e-4dce-95af-72a07ce53d56\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-xrnbx" Sep 11 20:01:31.622176 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:01:31.618375 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e40d01fe-6f2e-4dce-95af-72a07ce53d56-prometheus-operator-tls podName:e40d01fe-6f2e-4dce-95af-72a07ce53d56 nodeName:}" failed. No retries permitted until 2026-09-11 20:01:32.118350957 +0000 UTC m=+62.166401286 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/e40d01fe-6f2e-4dce-95af-72a07ce53d56-prometheus-operator-tls") pod "prometheus-operator-68bcd7f987-xrnbx" (UID: "e40d01fe-6f2e-4dce-95af-72a07ce53d56") : secret "prometheus-operator-tls" not found Sep 11 20:01:31.622176 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:31.618937 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e40d01fe-6f2e-4dce-95af-72a07ce53d56-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-xrnbx\" (UID: \"e40d01fe-6f2e-4dce-95af-72a07ce53d56\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-xrnbx" Sep 11 20:01:31.622176 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:31.620888 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e40d01fe-6f2e-4dce-95af-72a07ce53d56-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-xrnbx\" (UID: \"e40d01fe-6f2e-4dce-95af-72a07ce53d56\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-xrnbx" Sep 11 20:01:31.627390 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:31.627365 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc549\" (UniqueName: \"kubernetes.io/projected/e40d01fe-6f2e-4dce-95af-72a07ce53d56-kube-api-access-rc549\") pod \"prometheus-operator-68bcd7f987-xrnbx\" (UID: \"e40d01fe-6f2e-4dce-95af-72a07ce53d56\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-xrnbx" Sep 11 20:01:32.097790 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:32.097751 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:32.122550 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:32.122504 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e40d01fe-6f2e-4dce-95af-72a07ce53d56-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-xrnbx\" (UID: \"e40d01fe-6f2e-4dce-95af-72a07ce53d56\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-xrnbx" Sep 11 20:01:32.125255 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:32.125225 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e40d01fe-6f2e-4dce-95af-72a07ce53d56-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-xrnbx\" (UID: \"e40d01fe-6f2e-4dce-95af-72a07ce53d56\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-xrnbx" Sep 11 20:01:32.251339 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:32.251301 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:32.354595 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:32.354503 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:32.385494 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:32.385453 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-xrnbx" Sep 11 20:01:32.724380 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:32.724301 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-xrnbx"] Sep 11 20:01:32.727838 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:01:32.727811 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode40d01fe_6f2e_4dce_95af_72a07ce53d56.slice/crio-4f2cd75d54416f4458fed669c2cd2afc845e6c53deba62763b522b96ed0bce09 WatchSource:0}: Error finding container 4f2cd75d54416f4458fed669c2cd2afc845e6c53deba62763b522b96ed0bce09: Status 404 returned error can't find the container with id 4f2cd75d54416f4458fed669c2cd2afc845e6c53deba62763b522b96ed0bce09 Sep 11 20:01:32.749696 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:32.749667 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f4bb78b9b-gwm4b" event={"ID":"fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6","Type":"ContainerStarted","Data":"0fd808914c55514783b98ea07beb410cda9c55abcafb05b2d497c7f3562ebb0f"} Sep 11 20:01:32.750896 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:32.750870 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-xrnbx" event={"ID":"e40d01fe-6f2e-4dce-95af-72a07ce53d56","Type":"ContainerStarted","Data":"4f2cd75d54416f4458fed669c2cd2afc845e6c53deba62763b522b96ed0bce09"} Sep 11 20:01:32.771349 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:32.771300 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5f4bb78b9b-gwm4b" podStartSLOduration=1.515746837 podStartE2EDuration="4.771284582s" totalImagesPullingTime="3.255537745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:01:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:01:32.77021597 +0000 UTC m=+62.818266314" watchObservedRunningTime="2026-09-11 20:01:32.771284582 +0000 UTC m=+62.819334925" Sep 11 20:01:33.059390 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:33.059297 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:33.157958 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:33.157927 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:34.226788 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:34.226702 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:34.758843 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:34.758808 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-xrnbx" event={"ID":"e40d01fe-6f2e-4dce-95af-72a07ce53d56","Type":"ContainerStarted","Data":"48bfbc622167f41f5a48bb3f23ce04440da5225e5d66ef4af2dd2b7eb186a1af"} Sep 11 20:01:34.759017 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:34.758860 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-xrnbx" event={"ID":"e40d01fe-6f2e-4dce-95af-72a07ce53d56","Type":"ContainerStarted","Data":"b88476ad0d7d7087f519e1b97211036c768c7f6e71a350a6504701d442ecdaab"} Sep 11 20:01:34.775481 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:34.775434 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-xrnbx" podStartSLOduration=2.562079263 podStartE2EDuration="3.775417396s" totalImagesPullingTime="1.213338133s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:01:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:01:34.77519886 +0000 UTC m=+64.823249203" watchObservedRunningTime="2026-09-11 20:01:34.775417396 +0000 UTC m=+64.823467741" Sep 11 20:01:35.720015 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:35.719986 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rvgwn" Sep 11 20:01:36.052901 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:36.052816 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:36.155152 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:36.155111 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1f86594f-112f-4e58-a2ca-bbd951c4d1af-original-pull-secret\") pod \"global-pull-secret-syncer-q2dm2\" (UID: \"1f86594f-112f-4e58-a2ca-bbd951c4d1af\") " pod="kube-system/global-pull-secret-syncer-q2dm2" Sep 11 20:01:36.155152 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:36.155151 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwhtf\" (UniqueName: \"kubernetes.io/projected/ae2b5d1f-4bb1-41eb-9096-d54b304ad672-kube-api-access-cwhtf\") pod \"network-check-target-wk9hk\" (UID: \"ae2b5d1f-4bb1-41eb-9096-d54b304ad672\") " pod="openshift-network-diagnostics/network-check-target-wk9hk" Sep 11 20:01:36.157357 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:36.157331 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 11 20:01:36.157477 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:36.157388 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 11 20:01:36.167732 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:36.167711 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 11 20:01:36.168321 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:36.168302 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1f86594f-112f-4e58-a2ca-bbd951c4d1af-original-pull-secret\") pod \"global-pull-secret-syncer-q2dm2\" (UID: \"1f86594f-112f-4e58-a2ca-bbd951c4d1af\") " pod="kube-system/global-pull-secret-syncer-q2dm2" Sep 11 20:01:36.178340 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:36.178313 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwhtf\" (UniqueName: \"kubernetes.io/projected/ae2b5d1f-4bb1-41eb-9096-d54b304ad672-kube-api-access-cwhtf\") pod \"network-check-target-wk9hk\" (UID: \"ae2b5d1f-4bb1-41eb-9096-d54b304ad672\") " pod="openshift-network-diagnostics/network-check-target-wk9hk" Sep 11 20:01:36.256464 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:36.256425 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce96d552-27b5-4912-86b0-38f0f46d0002-metrics-certs\") pod \"network-metrics-daemon-zpl5q\" (UID: \"ce96d552-27b5-4912-86b0-38f0f46d0002\") " pod="openshift-multus/network-metrics-daemon-zpl5q" Sep 11 20:01:36.258627 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:36.258607 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 11 20:01:36.265266 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:36.265247 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2dm2" Sep 11 20:01:36.269434 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:36.269408 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce96d552-27b5-4912-86b0-38f0f46d0002-metrics-certs\") pod \"network-metrics-daemon-zpl5q\" (UID: \"ce96d552-27b5-4912-86b0-38f0f46d0002\") " pod="openshift-multus/network-metrics-daemon-zpl5q" Sep 11 20:01:36.272186 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:36.272165 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fv8gc\"" Sep 11 20:01:36.281031 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:36.281009 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wk9hk" Sep 11 20:01:36.558072 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:36.557997 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-b7zd7\"" Sep 11 20:01:36.564219 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:36.564177 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-q2dm2"] Sep 11 20:01:36.567079 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:36.567054 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zpl5q" Sep 11 20:01:36.567768 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:01:36.567747 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f86594f_112f_4e58_a2ca_bbd951c4d1af.slice/crio-7ddeb670f3c4d9d75635a6442a04a4d2bcf60cb4763b80b2e0001986eec8a7a4 WatchSource:0}: Error finding container 7ddeb670f3c4d9d75635a6442a04a4d2bcf60cb4763b80b2e0001986eec8a7a4: Status 404 returned error can't find the container with id 7ddeb670f3c4d9d75635a6442a04a4d2bcf60cb4763b80b2e0001986eec8a7a4 Sep 11 20:01:36.587196 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:36.587148 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wk9hk"] Sep 11 20:01:36.591772 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:01:36.591749 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae2b5d1f_4bb1_41eb_9096_d54b304ad672.slice/crio-0752ce9c8b3bcd264d1b1b882c4aa3f5268b93066bccaed91a481eb06af1f40e WatchSource:0}: Error finding container 0752ce9c8b3bcd264d1b1b882c4aa3f5268b93066bccaed91a481eb06af1f40e: Status 404 returned error can't find the container with id 0752ce9c8b3bcd264d1b1b882c4aa3f5268b93066bccaed91a481eb06af1f40e Sep 11 20:01:36.593869 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:36.593839 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:36.737453 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:36.737419 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:36.770778 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:36.770739 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wk9hk" event={"ID":"ae2b5d1f-4bb1-41eb-9096-d54b304ad672","Type":"ContainerStarted","Data":"0752ce9c8b3bcd264d1b1b882c4aa3f5268b93066bccaed91a481eb06af1f40e"} Sep 11 20:01:36.771821 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:36.771798 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-q2dm2" event={"ID":"1f86594f-112f-4e58-a2ca-bbd951c4d1af","Type":"ContainerStarted","Data":"7ddeb670f3c4d9d75635a6442a04a4d2bcf60cb4763b80b2e0001986eec8a7a4"} Sep 11 20:01:36.837187 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:36.837112 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:36.863040 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:36.863008 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zpl5q"] Sep 11 20:01:36.867842 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:01:36.867815 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce96d552_27b5_4912_86b0_38f0f46d0002.slice/crio-7c5b449a2a2900ee173660f10900202b0ef1af05a8c0f9f6fe6ddc3075a08031 WatchSource:0}: Error finding container 7c5b449a2a2900ee173660f10900202b0ef1af05a8c0f9f6fe6ddc3075a08031: Status 404 returned error can't find the container with id 7c5b449a2a2900ee173660f10900202b0ef1af05a8c0f9f6fe6ddc3075a08031 Sep 11 20:01:36.877672 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:36.877649 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9x7w5"] Sep 11 20:01:36.912447 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:36.912415 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9x7w5" Sep 11 20:01:36.914669 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:36.914641 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 11 20:01:36.914812 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:36.914654 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6rwxf\"" Sep 11 20:01:36.914812 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:36.914655 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 11 20:01:36.914971 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:36.914957 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 11 20:01:36.946917 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:36.946890 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:36.964033 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:36.964002 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d2d5f666-660a-47aa-85c7-4ba6e485c395-node-exporter-accelerators-collector-config\") pod \"node-exporter-9x7w5\" (UID: \"d2d5f666-660a-47aa-85c7-4ba6e485c395\") " pod="openshift-monitoring/node-exporter-9x7w5" Sep 11 20:01:36.964166 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:36.964045 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d2d5f666-660a-47aa-85c7-4ba6e485c395-node-exporter-wtmp\") pod \"node-exporter-9x7w5\" (UID: \"d2d5f666-660a-47aa-85c7-4ba6e485c395\") " pod="openshift-monitoring/node-exporter-9x7w5" Sep 11 20:01:36.964166 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:36.964075 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d2d5f666-660a-47aa-85c7-4ba6e485c395-root\") pod \"node-exporter-9x7w5\" (UID: \"d2d5f666-660a-47aa-85c7-4ba6e485c395\") " pod="openshift-monitoring/node-exporter-9x7w5" Sep 11 20:01:36.964166 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:36.964101 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d2d5f666-660a-47aa-85c7-4ba6e485c395-metrics-client-ca\") pod \"node-exporter-9x7w5\" (UID: \"d2d5f666-660a-47aa-85c7-4ba6e485c395\") " pod="openshift-monitoring/node-exporter-9x7w5" Sep 11 20:01:36.964166 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:36.964138 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grdz7\" (UniqueName: \"kubernetes.io/projected/d2d5f666-660a-47aa-85c7-4ba6e485c395-kube-api-access-grdz7\") pod \"node-exporter-9x7w5\" (UID: \"d2d5f666-660a-47aa-85c7-4ba6e485c395\") " pod="openshift-monitoring/node-exporter-9x7w5" Sep 11 20:01:36.964354 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:36.964182 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d2d5f666-660a-47aa-85c7-4ba6e485c395-sys\") pod \"node-exporter-9x7w5\" (UID: \"d2d5f666-660a-47aa-85c7-4ba6e485c395\") " pod="openshift-monitoring/node-exporter-9x7w5" Sep 11 20:01:36.964354 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:36.964204 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d2d5f666-660a-47aa-85c7-4ba6e485c395-node-exporter-tls\") pod \"node-exporter-9x7w5\" (UID: \"d2d5f666-660a-47aa-85c7-4ba6e485c395\") " pod="openshift-monitoring/node-exporter-9x7w5" Sep 11 20:01:36.964354 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:36.964229 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d2d5f666-660a-47aa-85c7-4ba6e485c395-node-exporter-textfile\") pod \"node-exporter-9x7w5\" (UID: \"d2d5f666-660a-47aa-85c7-4ba6e485c395\") " pod="openshift-monitoring/node-exporter-9x7w5" Sep 11 20:01:36.964354 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:36.964263 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d2d5f666-660a-47aa-85c7-4ba6e485c395-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9x7w5\" (UID: \"d2d5f666-660a-47aa-85c7-4ba6e485c395\") " pod="openshift-monitoring/node-exporter-9x7w5" Sep 11 20:01:37.034497 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:37.034462 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:37.066739 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:37.065451 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d2d5f666-660a-47aa-85c7-4ba6e485c395-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9x7w5\" (UID: \"d2d5f666-660a-47aa-85c7-4ba6e485c395\") " pod="openshift-monitoring/node-exporter-9x7w5" Sep 11 20:01:37.066739 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:37.065507 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d2d5f666-660a-47aa-85c7-4ba6e485c395-node-exporter-accelerators-collector-config\") pod \"node-exporter-9x7w5\" (UID: \"d2d5f666-660a-47aa-85c7-4ba6e485c395\") " pod="openshift-monitoring/node-exporter-9x7w5" Sep 11 20:01:37.066739 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:37.065548 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d2d5f666-660a-47aa-85c7-4ba6e485c395-node-exporter-wtmp\") pod \"node-exporter-9x7w5\" (UID: \"d2d5f666-660a-47aa-85c7-4ba6e485c395\") " pod="openshift-monitoring/node-exporter-9x7w5" Sep 11 20:01:37.066739 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:37.065596 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d2d5f666-660a-47aa-85c7-4ba6e485c395-root\") pod \"node-exporter-9x7w5\" (UID: \"d2d5f666-660a-47aa-85c7-4ba6e485c395\") " pod="openshift-monitoring/node-exporter-9x7w5" Sep 11 20:01:37.066739 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:37.065629 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d2d5f666-660a-47aa-85c7-4ba6e485c395-metrics-client-ca\") pod \"node-exporter-9x7w5\" (UID: \"d2d5f666-660a-47aa-85c7-4ba6e485c395\") " pod="openshift-monitoring/node-exporter-9x7w5" Sep 11 20:01:37.066739 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:37.065656 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grdz7\" (UniqueName: \"kubernetes.io/projected/d2d5f666-660a-47aa-85c7-4ba6e485c395-kube-api-access-grdz7\") pod \"node-exporter-9x7w5\" (UID: \"d2d5f666-660a-47aa-85c7-4ba6e485c395\") " pod="openshift-monitoring/node-exporter-9x7w5" Sep 11 20:01:37.066739 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:37.065710 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d2d5f666-660a-47aa-85c7-4ba6e485c395-sys\") pod \"node-exporter-9x7w5\" (UID: \"d2d5f666-660a-47aa-85c7-4ba6e485c395\") " pod="openshift-monitoring/node-exporter-9x7w5" Sep 11 20:01:37.066739 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:37.065748 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d2d5f666-660a-47aa-85c7-4ba6e485c395-node-exporter-tls\") pod \"node-exporter-9x7w5\" (UID: \"d2d5f666-660a-47aa-85c7-4ba6e485c395\") " pod="openshift-monitoring/node-exporter-9x7w5" Sep 11 20:01:37.066739 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:37.065773 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d2d5f666-660a-47aa-85c7-4ba6e485c395-node-exporter-textfile\") pod \"node-exporter-9x7w5\" (UID: \"d2d5f666-660a-47aa-85c7-4ba6e485c395\") " pod="openshift-monitoring/node-exporter-9x7w5" Sep 11 20:01:37.066739 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:37.066113 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d2d5f666-660a-47aa-85c7-4ba6e485c395-node-exporter-textfile\") pod \"node-exporter-9x7w5\" (UID: \"d2d5f666-660a-47aa-85c7-4ba6e485c395\") " pod="openshift-monitoring/node-exporter-9x7w5" Sep 11 20:01:37.066739 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:37.066499 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d2d5f666-660a-47aa-85c7-4ba6e485c395-sys\") pod \"node-exporter-9x7w5\" (UID: \"d2d5f666-660a-47aa-85c7-4ba6e485c395\") " pod="openshift-monitoring/node-exporter-9x7w5" Sep 11 20:01:37.067394 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:37.066933 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d2d5f666-660a-47aa-85c7-4ba6e485c395-metrics-client-ca\") pod \"node-exporter-9x7w5\" (UID: \"d2d5f666-660a-47aa-85c7-4ba6e485c395\") " pod="openshift-monitoring/node-exporter-9x7w5" Sep 11 20:01:37.067394 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:37.067306 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d2d5f666-660a-47aa-85c7-4ba6e485c395-node-exporter-accelerators-collector-config\") pod \"node-exporter-9x7w5\" (UID: \"d2d5f666-660a-47aa-85c7-4ba6e485c395\") " pod="openshift-monitoring/node-exporter-9x7w5" Sep 11 20:01:37.067489 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:37.067415 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d2d5f666-660a-47aa-85c7-4ba6e485c395-node-exporter-wtmp\") pod \"node-exporter-9x7w5\" (UID: \"d2d5f666-660a-47aa-85c7-4ba6e485c395\") " pod="openshift-monitoring/node-exporter-9x7w5" Sep 11 20:01:37.067489 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:37.067461 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d2d5f666-660a-47aa-85c7-4ba6e485c395-root\") pod \"node-exporter-9x7w5\" (UID: \"d2d5f666-660a-47aa-85c7-4ba6e485c395\") " pod="openshift-monitoring/node-exporter-9x7w5" Sep 11 20:01:37.071010 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:37.070959 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d2d5f666-660a-47aa-85c7-4ba6e485c395-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9x7w5\" (UID: \"d2d5f666-660a-47aa-85c7-4ba6e485c395\") " pod="openshift-monitoring/node-exporter-9x7w5" Sep 11 20:01:37.093165 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:37.093077 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d2d5f666-660a-47aa-85c7-4ba6e485c395-node-exporter-tls\") pod \"node-exporter-9x7w5\" (UID: \"d2d5f666-660a-47aa-85c7-4ba6e485c395\") " pod="openshift-monitoring/node-exporter-9x7w5" Sep 11 20:01:37.097504 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:37.097476 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grdz7\" (UniqueName: \"kubernetes.io/projected/d2d5f666-660a-47aa-85c7-4ba6e485c395-kube-api-access-grdz7\") pod \"node-exporter-9x7w5\" (UID: \"d2d5f666-660a-47aa-85c7-4ba6e485c395\") " pod="openshift-monitoring/node-exporter-9x7w5" Sep 11 20:01:37.153718 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:37.153688 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:37.221997 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:37.221959 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9x7w5" Sep 11 20:01:37.243832 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:01:37.243795 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2d5f666_660a_47aa_85c7_4ba6e485c395.slice/crio-cce92ca4748fb29956c6583a8bde50cac038bb04195c04cbb240a662ed46d626 WatchSource:0}: Error finding container cce92ca4748fb29956c6583a8bde50cac038bb04195c04cbb240a662ed46d626: Status 404 returned error can't find the container with id cce92ca4748fb29956c6583a8bde50cac038bb04195c04cbb240a662ed46d626 Sep 11 20:01:37.345442 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:37.345327 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:37.778116 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:37.778018 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zpl5q" event={"ID":"ce96d552-27b5-4912-86b0-38f0f46d0002","Type":"ContainerStarted","Data":"7c5b449a2a2900ee173660f10900202b0ef1af05a8c0f9f6fe6ddc3075a08031"} Sep 11 20:01:37.779703 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:37.779675 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9x7w5" event={"ID":"d2d5f666-660a-47aa-85c7-4ba6e485c395","Type":"ContainerStarted","Data":"cce92ca4748fb29956c6583a8bde50cac038bb04195c04cbb240a662ed46d626"} Sep 11 20:01:37.840560 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:37.840472 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:37.963157 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:37.961417 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 20:01:37.963157 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:37.962636 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:37.964981 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:37.964956 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:01:37.968781 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:37.967422 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 11 20:01:37.968781 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:37.967791 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-p4rwx\"" Sep 11 20:01:37.968781 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:37.967979 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 11 20:01:37.968781 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:37.968108 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 11 20:01:37.968781 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:37.968192 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 11 20:01:37.968781 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:37.968328 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 11 20:01:37.968781 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:37.968429 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 11 20:01:37.968781 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:37.968504 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 11 20:01:37.968781 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:37.968647 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 11 20:01:37.969495 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:37.969346 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 11 20:01:37.980256 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:37.980227 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 20:01:38.077090 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.077009 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ddf54b53-5ff6-47d2-bfb1-260649d17ecb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:01:38.077090 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.077061 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2g4s\" (UniqueName: \"kubernetes.io/projected/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-kube-api-access-f2g4s\") pod \"alertmanager-main-0\" (UID: \"ddf54b53-5ff6-47d2-bfb1-260649d17ecb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:01:38.077309 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.077119 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ddf54b53-5ff6-47d2-bfb1-260649d17ecb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:01:38.077309 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.077152 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ddf54b53-5ff6-47d2-bfb1-260649d17ecb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:01:38.077309 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.077207 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-config-out\") pod \"alertmanager-main-0\" (UID: \"ddf54b53-5ff6-47d2-bfb1-260649d17ecb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:01:38.077309 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.077244 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-web-config\") pod \"alertmanager-main-0\" (UID: \"ddf54b53-5ff6-47d2-bfb1-260649d17ecb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:01:38.077309 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.077263 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ddf54b53-5ff6-47d2-bfb1-260649d17ecb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:01:38.077309 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.077287 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ddf54b53-5ff6-47d2-bfb1-260649d17ecb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:01:38.077309 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.077302 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ddf54b53-5ff6-47d2-bfb1-260649d17ecb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:01:38.077683 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.077335 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ddf54b53-5ff6-47d2-bfb1-260649d17ecb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:01:38.077683 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.077376 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ddf54b53-5ff6-47d2-bfb1-260649d17ecb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:01:38.077683 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.077408 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ddf54b53-5ff6-47d2-bfb1-260649d17ecb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:01:38.077683 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.077439 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-config-volume\") pod \"alertmanager-main-0\" (UID: \"ddf54b53-5ff6-47d2-bfb1-260649d17ecb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:01:38.153301 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.153246 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:38.178608 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.178558 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ddf54b53-5ff6-47d2-bfb1-260649d17ecb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:01:38.178795 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.178628 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-config-out\") pod \"alertmanager-main-0\" (UID: \"ddf54b53-5ff6-47d2-bfb1-260649d17ecb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:01:38.178795 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.178666 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-web-config\") pod \"alertmanager-main-0\" (UID: \"ddf54b53-5ff6-47d2-bfb1-260649d17ecb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:01:38.178795 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.178694 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ddf54b53-5ff6-47d2-bfb1-260649d17ecb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:01:38.178795 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.178727 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ddf54b53-5ff6-47d2-bfb1-260649d17ecb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:01:38.178795 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.178752 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ddf54b53-5ff6-47d2-bfb1-260649d17ecb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:01:38.178795 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.178775 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ddf54b53-5ff6-47d2-bfb1-260649d17ecb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:01:38.179113 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.178809 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ddf54b53-5ff6-47d2-bfb1-260649d17ecb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:01:38.179113 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.178841 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ddf54b53-5ff6-47d2-bfb1-260649d17ecb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:01:38.179113 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.178871 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-config-volume\") pod \"alertmanager-main-0\" (UID: \"ddf54b53-5ff6-47d2-bfb1-260649d17ecb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:01:38.179113 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.178913 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ddf54b53-5ff6-47d2-bfb1-260649d17ecb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:01:38.179113 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.178940 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2g4s\" (UniqueName: \"kubernetes.io/projected/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-kube-api-access-f2g4s\") pod \"alertmanager-main-0\" (UID: \"ddf54b53-5ff6-47d2-bfb1-260649d17ecb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:01:38.179113 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.178962 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ddf54b53-5ff6-47d2-bfb1-260649d17ecb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:01:38.179461 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.179437 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ddf54b53-5ff6-47d2-bfb1-260649d17ecb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:01:38.180007 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.179980 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ddf54b53-5ff6-47d2-bfb1-260649d17ecb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:01:38.181243 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.181185 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ddf54b53-5ff6-47d2-bfb1-260649d17ecb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:01:38.183252 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.182634 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ddf54b53-5ff6-47d2-bfb1-260649d17ecb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:01:38.183252 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.183241 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ddf54b53-5ff6-47d2-bfb1-260649d17ecb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:01:38.185257 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.184536 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ddf54b53-5ff6-47d2-bfb1-260649d17ecb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:01:38.185257 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.184826 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-web-config\") pod \"alertmanager-main-0\" (UID: \"ddf54b53-5ff6-47d2-bfb1-260649d17ecb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:01:38.185257 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.184900 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-config-volume\") pod \"alertmanager-main-0\" (UID: \"ddf54b53-5ff6-47d2-bfb1-260649d17ecb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:01:38.185257 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.185069 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-config-out\") pod \"alertmanager-main-0\" (UID: \"ddf54b53-5ff6-47d2-bfb1-260649d17ecb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:01:38.185257 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.185219 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ddf54b53-5ff6-47d2-bfb1-260649d17ecb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:01:38.186557 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.186500 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ddf54b53-5ff6-47d2-bfb1-260649d17ecb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:01:38.186767 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.186749 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ddf54b53-5ff6-47d2-bfb1-260649d17ecb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:01:38.193180 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.193154 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2g4s\" (UniqueName: \"kubernetes.io/projected/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-kube-api-access-f2g4s\") pod \"alertmanager-main-0\" (UID: \"ddf54b53-5ff6-47d2-bfb1-260649d17ecb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:01:38.281152 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.281039 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:01:38.703108 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.703078 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 20:01:38.710723 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:01:38.710688 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddf54b53_5ff6_47d2_bfb1_260649d17ecb.slice/crio-704accae14af3cbfe78f50bc27d4a331aed886998c7847cbb29cdb83d40076b0 WatchSource:0}: Error finding container 704accae14af3cbfe78f50bc27d4a331aed886998c7847cbb29cdb83d40076b0: Status 404 returned error can't find the container with id 704accae14af3cbfe78f50bc27d4a331aed886998c7847cbb29cdb83d40076b0 Sep 11 20:01:38.786298 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.786261 2546 generic.go:356] "Generic (PLEG): container finished" podID="d2d5f666-660a-47aa-85c7-4ba6e485c395" containerID="0fbf9026296411c19445e139abfd773ff85937d2a74fc8ecb6b0da74670a7c6a" exitCode=0 Sep 11 20:01:38.786767 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.786401 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9x7w5" event={"ID":"d2d5f666-660a-47aa-85c7-4ba6e485c395","Type":"ContainerDied","Data":"0fbf9026296411c19445e139abfd773ff85937d2a74fc8ecb6b0da74670a7c6a"} Sep 11 20:01:38.790415 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.790349 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zpl5q" event={"ID":"ce96d552-27b5-4912-86b0-38f0f46d0002","Type":"ContainerStarted","Data":"129d51b03ddf535048d4157beca57ace732cd28745debe818cdb8fb2007da519"} Sep 11 20:01:38.790415 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.790412 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zpl5q" event={"ID":"ce96d552-27b5-4912-86b0-38f0f46d0002","Type":"ContainerStarted","Data":"04fa8b2ee21ab8d9c44f576e0669af962ee79ce21c1556870e18b6e4902152c0"} Sep 11 20:01:38.792526 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.792447 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ddf54b53-5ff6-47d2-bfb1-260649d17ecb","Type":"ContainerStarted","Data":"704accae14af3cbfe78f50bc27d4a331aed886998c7847cbb29cdb83d40076b0"} Sep 11 20:01:38.825260 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.825232 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7b5b4c8886-wf7z8"] Sep 11 20:01:38.828631 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.828565 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-zpl5q" podStartSLOduration=67.587022117 podStartE2EDuration="1m8.828546737s" totalImagesPullingTime="1.24152462s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:00:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:01:38.826674947 +0000 UTC m=+68.874725314" watchObservedRunningTime="2026-09-11 20:01:38.828546737 +0000 UTC m=+68.876597082" Sep 11 20:01:38.831485 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.831462 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7b5b4c8886-wf7z8" Sep 11 20:01:38.833776 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.833756 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-333952dpk9qhq\"" Sep 11 20:01:38.833879 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.833779 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 11 20:01:38.834015 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.834000 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 11 20:01:38.834080 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.834029 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 11 20:01:38.834132 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.834104 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 11 20:01:38.834241 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.834226 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 11 20:01:38.834379 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.834363 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-6pkkp\"" Sep 11 20:01:38.842598 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.842558 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7b5b4c8886-wf7z8"] Sep 11 20:01:38.885414 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.885385 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0d6de2e0-f1ba-443b-9624-7144ab50bf04-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7b5b4c8886-wf7z8\" (UID: \"0d6de2e0-f1ba-443b-9624-7144ab50bf04\") " pod="openshift-monitoring/thanos-querier-7b5b4c8886-wf7z8" Sep 11 20:01:38.885521 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.885449 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0d6de2e0-f1ba-443b-9624-7144ab50bf04-secret-grpc-tls\") pod \"thanos-querier-7b5b4c8886-wf7z8\" (UID: \"0d6de2e0-f1ba-443b-9624-7144ab50bf04\") " pod="openshift-monitoring/thanos-querier-7b5b4c8886-wf7z8" Sep 11 20:01:38.885561 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.885536 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/0d6de2e0-f1ba-443b-9624-7144ab50bf04-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7b5b4c8886-wf7z8\" (UID: \"0d6de2e0-f1ba-443b-9624-7144ab50bf04\") " pod="openshift-monitoring/thanos-querier-7b5b4c8886-wf7z8" Sep 11 20:01:38.885618 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.885592 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0d6de2e0-f1ba-443b-9624-7144ab50bf04-secret-thanos-querier-tls\") pod \"thanos-querier-7b5b4c8886-wf7z8\" (UID: \"0d6de2e0-f1ba-443b-9624-7144ab50bf04\") " pod="openshift-monitoring/thanos-querier-7b5b4c8886-wf7z8" Sep 11 20:01:38.885680 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.885663 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcrgk\" (UniqueName: \"kubernetes.io/projected/0d6de2e0-f1ba-443b-9624-7144ab50bf04-kube-api-access-fcrgk\") pod \"thanos-querier-7b5b4c8886-wf7z8\" (UID: \"0d6de2e0-f1ba-443b-9624-7144ab50bf04\") " pod="openshift-monitoring/thanos-querier-7b5b4c8886-wf7z8" Sep 11 20:01:38.885730 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.885715 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0d6de2e0-f1ba-443b-9624-7144ab50bf04-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7b5b4c8886-wf7z8\" (UID: \"0d6de2e0-f1ba-443b-9624-7144ab50bf04\") " pod="openshift-monitoring/thanos-querier-7b5b4c8886-wf7z8" Sep 11 20:01:38.885814 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.885796 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0d6de2e0-f1ba-443b-9624-7144ab50bf04-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7b5b4c8886-wf7z8\" (UID: \"0d6de2e0-f1ba-443b-9624-7144ab50bf04\") " pod="openshift-monitoring/thanos-querier-7b5b4c8886-wf7z8" Sep 11 20:01:38.885873 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.885834 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d6de2e0-f1ba-443b-9624-7144ab50bf04-metrics-client-ca\") pod \"thanos-querier-7b5b4c8886-wf7z8\" (UID: \"0d6de2e0-f1ba-443b-9624-7144ab50bf04\") " pod="openshift-monitoring/thanos-querier-7b5b4c8886-wf7z8" Sep 11 20:01:38.943042 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.943012 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:38.962399 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.962364 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5f4bb78b9b-gwm4b" Sep 11 20:01:38.962559 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.962439 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5f4bb78b9b-gwm4b" Sep 11 20:01:38.968558 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.968514 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5f4bb78b9b-gwm4b" Sep 11 20:01:38.986568 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.986468 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/0d6de2e0-f1ba-443b-9624-7144ab50bf04-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7b5b4c8886-wf7z8\" (UID: \"0d6de2e0-f1ba-443b-9624-7144ab50bf04\") " pod="openshift-monitoring/thanos-querier-7b5b4c8886-wf7z8" Sep 11 20:01:38.986568 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.986524 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0d6de2e0-f1ba-443b-9624-7144ab50bf04-secret-thanos-querier-tls\") pod \"thanos-querier-7b5b4c8886-wf7z8\" (UID: \"0d6de2e0-f1ba-443b-9624-7144ab50bf04\") " pod="openshift-monitoring/thanos-querier-7b5b4c8886-wf7z8" Sep 11 20:01:38.986568 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.986553 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcrgk\" (UniqueName: \"kubernetes.io/projected/0d6de2e0-f1ba-443b-9624-7144ab50bf04-kube-api-access-fcrgk\") pod \"thanos-querier-7b5b4c8886-wf7z8\" (UID: \"0d6de2e0-f1ba-443b-9624-7144ab50bf04\") " pod="openshift-monitoring/thanos-querier-7b5b4c8886-wf7z8" Sep 11 20:01:38.986830 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.986610 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0d6de2e0-f1ba-443b-9624-7144ab50bf04-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7b5b4c8886-wf7z8\" (UID: \"0d6de2e0-f1ba-443b-9624-7144ab50bf04\") " pod="openshift-monitoring/thanos-querier-7b5b4c8886-wf7z8" Sep 11 20:01:38.987217 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.986903 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0d6de2e0-f1ba-443b-9624-7144ab50bf04-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7b5b4c8886-wf7z8\" (UID: \"0d6de2e0-f1ba-443b-9624-7144ab50bf04\") " pod="openshift-monitoring/thanos-querier-7b5b4c8886-wf7z8" Sep 11 20:01:38.987217 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.986951 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d6de2e0-f1ba-443b-9624-7144ab50bf04-metrics-client-ca\") pod \"thanos-querier-7b5b4c8886-wf7z8\" (UID: \"0d6de2e0-f1ba-443b-9624-7144ab50bf04\") " pod="openshift-monitoring/thanos-querier-7b5b4c8886-wf7z8" Sep 11 20:01:38.987217 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.987015 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0d6de2e0-f1ba-443b-9624-7144ab50bf04-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7b5b4c8886-wf7z8\" (UID: \"0d6de2e0-f1ba-443b-9624-7144ab50bf04\") " pod="openshift-monitoring/thanos-querier-7b5b4c8886-wf7z8" Sep 11 20:01:38.987217 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.987045 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0d6de2e0-f1ba-443b-9624-7144ab50bf04-secret-grpc-tls\") pod \"thanos-querier-7b5b4c8886-wf7z8\" (UID: \"0d6de2e0-f1ba-443b-9624-7144ab50bf04\") " pod="openshift-monitoring/thanos-querier-7b5b4c8886-wf7z8" Sep 11 20:01:38.987813 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.987756 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d6de2e0-f1ba-443b-9624-7144ab50bf04-metrics-client-ca\") pod \"thanos-querier-7b5b4c8886-wf7z8\" (UID: \"0d6de2e0-f1ba-443b-9624-7144ab50bf04\") " pod="openshift-monitoring/thanos-querier-7b5b4c8886-wf7z8" Sep 11 20:01:38.990006 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.989950 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0d6de2e0-f1ba-443b-9624-7144ab50bf04-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7b5b4c8886-wf7z8\" (UID: \"0d6de2e0-f1ba-443b-9624-7144ab50bf04\") " pod="openshift-monitoring/thanos-querier-7b5b4c8886-wf7z8" Sep 11 20:01:38.990604 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.990136 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0d6de2e0-f1ba-443b-9624-7144ab50bf04-secret-grpc-tls\") pod \"thanos-querier-7b5b4c8886-wf7z8\" (UID: \"0d6de2e0-f1ba-443b-9624-7144ab50bf04\") " pod="openshift-monitoring/thanos-querier-7b5b4c8886-wf7z8" Sep 11 20:01:38.990604 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.990211 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0d6de2e0-f1ba-443b-9624-7144ab50bf04-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7b5b4c8886-wf7z8\" (UID: \"0d6de2e0-f1ba-443b-9624-7144ab50bf04\") " pod="openshift-monitoring/thanos-querier-7b5b4c8886-wf7z8" Sep 11 20:01:38.990604 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.990543 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0d6de2e0-f1ba-443b-9624-7144ab50bf04-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7b5b4c8886-wf7z8\" (UID: \"0d6de2e0-f1ba-443b-9624-7144ab50bf04\") " pod="openshift-monitoring/thanos-querier-7b5b4c8886-wf7z8" Sep 11 20:01:38.991008 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.990985 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/0d6de2e0-f1ba-443b-9624-7144ab50bf04-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7b5b4c8886-wf7z8\" (UID: \"0d6de2e0-f1ba-443b-9624-7144ab50bf04\") " pod="openshift-monitoring/thanos-querier-7b5b4c8886-wf7z8" Sep 11 20:01:38.991751 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:38.991725 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0d6de2e0-f1ba-443b-9624-7144ab50bf04-secret-thanos-querier-tls\") pod \"thanos-querier-7b5b4c8886-wf7z8\" (UID: \"0d6de2e0-f1ba-443b-9624-7144ab50bf04\") " pod="openshift-monitoring/thanos-querier-7b5b4c8886-wf7z8" Sep 11 20:01:39.001862 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:39.001823 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcrgk\" (UniqueName: \"kubernetes.io/projected/0d6de2e0-f1ba-443b-9624-7144ab50bf04-kube-api-access-fcrgk\") pod \"thanos-querier-7b5b4c8886-wf7z8\" (UID: \"0d6de2e0-f1ba-443b-9624-7144ab50bf04\") " pod="openshift-monitoring/thanos-querier-7b5b4c8886-wf7z8" Sep 11 20:01:39.040649 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:39.040620 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:39.154526 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:39.154471 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7b5b4c8886-wf7z8" Sep 11 20:01:39.780094 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:39.780060 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:39.798913 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:39.798872 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9x7w5" event={"ID":"d2d5f666-660a-47aa-85c7-4ba6e485c395","Type":"ContainerStarted","Data":"cfb2088600c3738c72ce0874298f4897ea1608365fe4d759748706be780c3468"} Sep 11 20:01:39.799370 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:39.798940 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9x7w5" event={"ID":"d2d5f666-660a-47aa-85c7-4ba6e485c395","Type":"ContainerStarted","Data":"326618fa4b42a57376d6753b77674c300a4f1525a0d512cea2abc3a16037b2ed"} Sep 11 20:01:39.803459 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:39.803434 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5f4bb78b9b-gwm4b" Sep 11 20:01:39.822062 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:39.822004 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9x7w5" podStartSLOduration=2.508109899 podStartE2EDuration="3.821984912s" totalImagesPullingTime="1.230193987s" totalInitContainerRuntime="83.681026ms" isStatefulPod=true podCreationTimestamp="2026-09-11 20:01:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:01:39.821416045 +0000 UTC m=+69.869466390" watchObservedRunningTime="2026-09-11 20:01:39.821984912 +0000 UTC m=+69.870035255" Sep 11 20:01:39.879992 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:39.879964 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:39.980505 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:39.980465 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:40.043346 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:40.043227 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:40.105530 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:40.105427 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:40.289238 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:40.289201 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:41.649637 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:41.649599 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:41.697489 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:41.697430 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-66875d4f5b-gt7sc"] Sep 11 20:01:41.701803 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:41.701781 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66875d4f5b-gt7sc" Sep 11 20:01:41.710768 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:41.710735 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 11 20:01:41.714168 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:41.714147 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66875d4f5b-gt7sc"] Sep 11 20:01:41.737275 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:41.737239 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:41.810509 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:41.810478 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8dc9b50e-1594-425a-aa20-8f355bdf6d65-trusted-ca-bundle\") pod \"console-66875d4f5b-gt7sc\" (UID: \"8dc9b50e-1594-425a-aa20-8f355bdf6d65\") " pod="openshift-console/console-66875d4f5b-gt7sc" Sep 11 20:01:41.810628 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:41.810535 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svg6m\" (UniqueName: \"kubernetes.io/projected/8dc9b50e-1594-425a-aa20-8f355bdf6d65-kube-api-access-svg6m\") pod \"console-66875d4f5b-gt7sc\" (UID: \"8dc9b50e-1594-425a-aa20-8f355bdf6d65\") " pod="openshift-console/console-66875d4f5b-gt7sc" Sep 11 20:01:41.810628 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:41.810567 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8dc9b50e-1594-425a-aa20-8f355bdf6d65-oauth-serving-cert\") pod \"console-66875d4f5b-gt7sc\" (UID: \"8dc9b50e-1594-425a-aa20-8f355bdf6d65\") " pod="openshift-console/console-66875d4f5b-gt7sc" Sep 11 20:01:41.810736 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:41.810629 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8dc9b50e-1594-425a-aa20-8f355bdf6d65-service-ca\") pod \"console-66875d4f5b-gt7sc\" (UID: \"8dc9b50e-1594-425a-aa20-8f355bdf6d65\") " pod="openshift-console/console-66875d4f5b-gt7sc" Sep 11 20:01:41.810736 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:41.810659 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8dc9b50e-1594-425a-aa20-8f355bdf6d65-console-oauth-config\") pod \"console-66875d4f5b-gt7sc\" (UID: \"8dc9b50e-1594-425a-aa20-8f355bdf6d65\") " pod="openshift-console/console-66875d4f5b-gt7sc" Sep 11 20:01:41.810736 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:41.810693 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8dc9b50e-1594-425a-aa20-8f355bdf6d65-console-serving-cert\") pod \"console-66875d4f5b-gt7sc\" (UID: \"8dc9b50e-1594-425a-aa20-8f355bdf6d65\") " pod="openshift-console/console-66875d4f5b-gt7sc" Sep 11 20:01:41.810736 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:41.810716 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8dc9b50e-1594-425a-aa20-8f355bdf6d65-console-config\") pod \"console-66875d4f5b-gt7sc\" (UID: \"8dc9b50e-1594-425a-aa20-8f355bdf6d65\") " pod="openshift-console/console-66875d4f5b-gt7sc" Sep 11 20:01:41.837706 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:41.837679 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:41.912132 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:41.912098 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8dc9b50e-1594-425a-aa20-8f355bdf6d65-oauth-serving-cert\") pod \"console-66875d4f5b-gt7sc\" (UID: \"8dc9b50e-1594-425a-aa20-8f355bdf6d65\") " pod="openshift-console/console-66875d4f5b-gt7sc" Sep 11 20:01:41.912251 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:41.912167 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8dc9b50e-1594-425a-aa20-8f355bdf6d65-service-ca\") pod \"console-66875d4f5b-gt7sc\" (UID: \"8dc9b50e-1594-425a-aa20-8f355bdf6d65\") " pod="openshift-console/console-66875d4f5b-gt7sc" Sep 11 20:01:41.912251 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:41.912207 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8dc9b50e-1594-425a-aa20-8f355bdf6d65-console-oauth-config\") pod \"console-66875d4f5b-gt7sc\" (UID: \"8dc9b50e-1594-425a-aa20-8f355bdf6d65\") " pod="openshift-console/console-66875d4f5b-gt7sc" Sep 11 20:01:41.912251 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:41.912241 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8dc9b50e-1594-425a-aa20-8f355bdf6d65-console-serving-cert\") pod \"console-66875d4f5b-gt7sc\" (UID: \"8dc9b50e-1594-425a-aa20-8f355bdf6d65\") " pod="openshift-console/console-66875d4f5b-gt7sc" Sep 11 20:01:41.912472 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:41.912269 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8dc9b50e-1594-425a-aa20-8f355bdf6d65-console-config\") pod \"console-66875d4f5b-gt7sc\" (UID: \"8dc9b50e-1594-425a-aa20-8f355bdf6d65\") " pod="openshift-console/console-66875d4f5b-gt7sc" Sep 11 20:01:41.912472 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:41.912326 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8dc9b50e-1594-425a-aa20-8f355bdf6d65-trusted-ca-bundle\") pod \"console-66875d4f5b-gt7sc\" (UID: \"8dc9b50e-1594-425a-aa20-8f355bdf6d65\") " pod="openshift-console/console-66875d4f5b-gt7sc" Sep 11 20:01:41.912472 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:41.912365 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svg6m\" (UniqueName: \"kubernetes.io/projected/8dc9b50e-1594-425a-aa20-8f355bdf6d65-kube-api-access-svg6m\") pod \"console-66875d4f5b-gt7sc\" (UID: \"8dc9b50e-1594-425a-aa20-8f355bdf6d65\") " pod="openshift-console/console-66875d4f5b-gt7sc" Sep 11 20:01:41.912689 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:41.912670 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8dc9b50e-1594-425a-aa20-8f355bdf6d65-oauth-serving-cert\") pod \"console-66875d4f5b-gt7sc\" (UID: \"8dc9b50e-1594-425a-aa20-8f355bdf6d65\") " pod="openshift-console/console-66875d4f5b-gt7sc" Sep 11 20:01:41.913361 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:41.913340 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8dc9b50e-1594-425a-aa20-8f355bdf6d65-service-ca\") pod \"console-66875d4f5b-gt7sc\" (UID: \"8dc9b50e-1594-425a-aa20-8f355bdf6d65\") " pod="openshift-console/console-66875d4f5b-gt7sc" Sep 11 20:01:41.913361 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:41.913346 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8dc9b50e-1594-425a-aa20-8f355bdf6d65-console-config\") pod \"console-66875d4f5b-gt7sc\" (UID: \"8dc9b50e-1594-425a-aa20-8f355bdf6d65\") " pod="openshift-console/console-66875d4f5b-gt7sc" Sep 11 20:01:41.913566 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:41.913547 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8dc9b50e-1594-425a-aa20-8f355bdf6d65-trusted-ca-bundle\") pod \"console-66875d4f5b-gt7sc\" (UID: \"8dc9b50e-1594-425a-aa20-8f355bdf6d65\") " pod="openshift-console/console-66875d4f5b-gt7sc" Sep 11 20:01:41.914790 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:41.914770 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8dc9b50e-1594-425a-aa20-8f355bdf6d65-console-serving-cert\") pod \"console-66875d4f5b-gt7sc\" (UID: \"8dc9b50e-1594-425a-aa20-8f355bdf6d65\") " pod="openshift-console/console-66875d4f5b-gt7sc" Sep 11 20:01:41.915279 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:41.915264 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8dc9b50e-1594-425a-aa20-8f355bdf6d65-console-oauth-config\") pod \"console-66875d4f5b-gt7sc\" (UID: \"8dc9b50e-1594-425a-aa20-8f355bdf6d65\") " pod="openshift-console/console-66875d4f5b-gt7sc" Sep 11 20:01:41.919673 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:41.919648 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svg6m\" (UniqueName: \"kubernetes.io/projected/8dc9b50e-1594-425a-aa20-8f355bdf6d65-kube-api-access-svg6m\") pod \"console-66875d4f5b-gt7sc\" (UID: \"8dc9b50e-1594-425a-aa20-8f355bdf6d65\") " pod="openshift-console/console-66875d4f5b-gt7sc" Sep 11 20:01:42.013547 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:42.013456 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66875d4f5b-gt7sc" Sep 11 20:01:42.027804 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:42.027774 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7b5b4c8886-wf7z8"] Sep 11 20:01:42.030485 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:01:42.030457 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d6de2e0_f1ba_443b_9624_7144ab50bf04.slice/crio-36f6eb71721138d3ddaaa852fb21ef9393ac291ee57a7ad1f83703c77169b616 WatchSource:0}: Error finding container 36f6eb71721138d3ddaaa852fb21ef9393ac291ee57a7ad1f83703c77169b616: Status 404 returned error can't find the container with id 36f6eb71721138d3ddaaa852fb21ef9393ac291ee57a7ad1f83703c77169b616 Sep 11 20:01:42.286634 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:42.286547 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66875d4f5b-gt7sc"] Sep 11 20:01:42.289679 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:01:42.289652 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dc9b50e_1594_425a_aa20_8f355bdf6d65.slice/crio-877674da9e7846f09647273c04a3925a9280a154d137f7ecae1ed1da4b01700d WatchSource:0}: Error finding container 877674da9e7846f09647273c04a3925a9280a154d137f7ecae1ed1da4b01700d: Status 404 returned error can't find the container with id 877674da9e7846f09647273c04a3925a9280a154d137f7ecae1ed1da4b01700d Sep 11 20:01:42.822464 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:42.822424 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66875d4f5b-gt7sc" event={"ID":"8dc9b50e-1594-425a-aa20-8f355bdf6d65","Type":"ContainerStarted","Data":"22bea90f988f3958b0110a81e95149e6bbd8662969b8ecea0d9f32d40fee4427"} Sep 11 20:01:42.822971 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:42.822483 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66875d4f5b-gt7sc" event={"ID":"8dc9b50e-1594-425a-aa20-8f355bdf6d65","Type":"ContainerStarted","Data":"877674da9e7846f09647273c04a3925a9280a154d137f7ecae1ed1da4b01700d"} Sep 11 20:01:42.824184 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:42.824139 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wk9hk" event={"ID":"ae2b5d1f-4bb1-41eb-9096-d54b304ad672","Type":"ContainerStarted","Data":"4bf4aa2f6970093cdca01e10f7d96d9011d15901346787ca481c3934be945ea6"} Sep 11 20:01:42.824305 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:42.824242 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wk9hk" Sep 11 20:01:42.827202 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:42.827170 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-q2dm2" event={"ID":"1f86594f-112f-4e58-a2ca-bbd951c4d1af","Type":"ContainerStarted","Data":"5a2666f1ff0534b00e472bc5684a387276321ce8b2288b6eb16ba7a5a07360cf"} Sep 11 20:01:42.828962 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:42.828938 2546 generic.go:356] "Generic (PLEG): container finished" podID="ddf54b53-5ff6-47d2-bfb1-260649d17ecb" containerID="0af2d149007812e42ac45dca18739f71b1642eeac5cffc77d81c37da836723ae" exitCode=0 Sep 11 20:01:42.829067 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:42.828965 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ddf54b53-5ff6-47d2-bfb1-260649d17ecb","Type":"ContainerDied","Data":"0af2d149007812e42ac45dca18739f71b1642eeac5cffc77d81c37da836723ae"} Sep 11 20:01:42.830181 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:42.830159 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b5b4c8886-wf7z8" event={"ID":"0d6de2e0-f1ba-443b-9624-7144ab50bf04","Type":"ContainerStarted","Data":"36f6eb71721138d3ddaaa852fb21ef9393ac291ee57a7ad1f83703c77169b616"} Sep 11 20:01:42.841636 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:42.841597 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-66875d4f5b-gt7sc" podStartSLOduration=1.841568524 podStartE2EDuration="1.841568524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:01:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:01:42.840760649 +0000 UTC m=+72.888810996" watchObservedRunningTime="2026-09-11 20:01:42.841568524 +0000 UTC m=+72.889618867" Sep 11 20:01:42.885502 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:42.885439 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-wk9hk" podStartSLOduration=67.732973349 podStartE2EDuration="1m12.885419193s" totalImagesPullingTime="5.152445844s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:00:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:01:42.884594472 +0000 UTC m=+72.932644819" watchObservedRunningTime="2026-09-11 20:01:42.885419193 +0000 UTC m=+72.933469538" Sep 11 20:01:42.898738 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:42.898684 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-q2dm2" podStartSLOduration=67.709090403 podStartE2EDuration="1m12.898664804s" totalImagesPullingTime="5.189574401s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 20:00:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:01:42.898246702 +0000 UTC m=+72.946297048" watchObservedRunningTime="2026-09-11 20:01:42.898664804 +0000 UTC m=+72.946715150" Sep 11 20:01:43.065386 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.065346 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:43.072013 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.071981 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 20:01:43.076147 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.076089 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:01:43.083779 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.083527 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 11 20:01:43.084189 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.084169 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 11 20:01:43.084277 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.084256 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 11 20:01:43.085167 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.084491 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 11 20:01:43.085167 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.084710 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 11 20:01:43.085167 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.084887 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 11 20:01:43.085167 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.084997 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 11 20:01:43.085167 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.085063 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 11 20:01:43.085799 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.085214 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 11 20:01:43.085799 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.085419 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 11 20:01:43.085799 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.085601 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 11 20:01:43.085799 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.085671 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 11 20:01:43.085799 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.085737 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 11 20:01:43.086050 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.085885 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-4mtrh\"" Sep 11 20:01:43.086050 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.085988 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 11 20:01:43.086152 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.086062 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-a8bpa4l5b72tl\"" Sep 11 20:01:43.087745 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.087713 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 11 20:01:43.105608 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.105468 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 20:01:43.121724 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.121696 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:01:43.121866 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.121745 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:01:43.121866 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.121786 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltlbl\" (UniqueName: \"kubernetes.io/projected/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-kube-api-access-ltlbl\") pod \"prometheus-k8s-0\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:01:43.121866 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.121821 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:01:43.121866 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.121846 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-config-out\") pod \"prometheus-k8s-0\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:01:43.122072 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.121866 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:01:43.122072 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.121900 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:01:43.122072 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.121922 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:01:43.122072 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.121953 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:01:43.122072 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.121983 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:01:43.122072 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.122008 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:01:43.122072 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.122039 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:01:43.122072 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.122073 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:01:43.122327 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.122095 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:01:43.122327 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.122122 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-config\") pod \"prometheus-k8s-0\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:01:43.122327 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.122234 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-web-config\") pod \"prometheus-k8s-0\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:01:43.122327 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.122271 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:01:43.122499 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.122330 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:01:43.122499 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.122444 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:01:43.122499 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.122479 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:01:43.154486 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.154452 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:43.223193 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.223096 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltlbl\" (UniqueName: \"kubernetes.io/projected/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-kube-api-access-ltlbl\") pod \"prometheus-k8s-0\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:01:43.224030 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.223917 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:01:43.224151 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.223986 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-config-out\") pod \"prometheus-k8s-0\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:01:43.224233 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.224175 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:01:43.224233 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.224215 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:01:43.224335 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.224240 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:01:43.224335 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.224276 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:01:43.224335 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.224310 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:01:43.224505 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.224340 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:01:43.224505 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.224374 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:01:43.224505 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.224411 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:01:43.224505 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.224435 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:01:43.224505 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.224465 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-config\") pod \"prometheus-k8s-0\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:01:43.224505 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.224500 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-web-config\") pod \"prometheus-k8s-0\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:01:43.224833 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.224530 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:01:43.224833 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.224560 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:01:43.224833 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.224620 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:01:43.224833 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.224649 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:01:43.224833 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.224685 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:01:43.224833 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.224707 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:01:43.224833 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.224725 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:01:43.225158 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.225143 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:01:43.226434 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.225773 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:01:43.227262 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.227233 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:01:43.227367 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.227275 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:01:43.227367 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.227287 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:01:43.227502 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.227477 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-config-out\") pod \"prometheus-k8s-0\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:01:43.227997 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.227851 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:01:43.228665 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.228185 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:01:43.228665 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.228282 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:01:43.228830 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.228680 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:01:43.228830 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.228802 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:01:43.230166 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.230127 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:01:43.230400 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.230338 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-web-config\") pod \"prometheus-k8s-0\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:01:43.230400 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.230385 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:01:43.230878 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.230854 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:01:43.231237 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.231209 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-config\") pod \"prometheus-k8s-0\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:01:43.231337 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.231249 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:01:43.232144 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.232108 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:01:43.242418 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.239226 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltlbl\" (UniqueName: \"kubernetes.io/projected/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-kube-api-access-ltlbl\") pod \"prometheus-k8s-0\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:01:43.390773 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.390731 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:01:43.524363 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.524299 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:43.624138 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.624095 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:43.695939 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.695910 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 20:01:43.697716 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:01:43.697685 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71e7ab2f_2146_48a4_a9a0_b6ac3de11385.slice/crio-54ed21b2934e04938de9cf5f5ae400df1581e3f32d82cfa00c2b6892174bf990 WatchSource:0}: Error finding container 54ed21b2934e04938de9cf5f5ae400df1581e3f32d82cfa00c2b6892174bf990: Status 404 returned error can't find the container with id 54ed21b2934e04938de9cf5f5ae400df1581e3f32d82cfa00c2b6892174bf990 Sep 11 20:01:43.837117 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.837078 2546 generic.go:356] "Generic (PLEG): container finished" podID="71e7ab2f-2146-48a4-a9a0-b6ac3de11385" containerID="4d7aa46e3dc629a5238ca22dc98949c1627827ca573e5c5fba3b700cd5d6098b" exitCode=0 Sep 11 20:01:43.837529 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.837245 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"71e7ab2f-2146-48a4-a9a0-b6ac3de11385","Type":"ContainerDied","Data":"4d7aa46e3dc629a5238ca22dc98949c1627827ca573e5c5fba3b700cd5d6098b"} Sep 11 20:01:43.837529 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:43.837319 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"71e7ab2f-2146-48a4-a9a0-b6ac3de11385","Type":"ContainerStarted","Data":"54ed21b2934e04938de9cf5f5ae400df1581e3f32d82cfa00c2b6892174bf990"} Sep 11 20:01:44.066069 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:44.065983 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:44.128715 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:44.128645 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:44.872086 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:44.872062 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:44.982395 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:44.981677 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:45.848490 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:45.848457 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ddf54b53-5ff6-47d2-bfb1-260649d17ecb","Type":"ContainerStarted","Data":"fa78a576841dfd3bf0754b37bf8dc2ba4fb8330c32decf03631ff15f4f4dc8bd"} Sep 11 20:01:45.848670 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:45.848570 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ddf54b53-5ff6-47d2-bfb1-260649d17ecb","Type":"ContainerStarted","Data":"af0c29c4fa1e3e932c8dbf144e67c942833eed221b6f79ef06edb06dbc453766"} Sep 11 20:01:45.848670 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:45.848623 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ddf54b53-5ff6-47d2-bfb1-260649d17ecb","Type":"ContainerStarted","Data":"47e36fec62fb15fc3ecbc383518bbead193e1cf4de3c5c2baeb44ddf70f93421"} Sep 11 20:01:45.848670 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:45.848648 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ddf54b53-5ff6-47d2-bfb1-260649d17ecb","Type":"ContainerStarted","Data":"91d34e9fafd743d1cc62cc58f05b7da54aacdcfd87137e9483732b05c6934dd1"} Sep 11 20:01:45.848670 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:45.848671 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ddf54b53-5ff6-47d2-bfb1-260649d17ecb","Type":"ContainerStarted","Data":"3be599218d92c14d9973ffa3cd4f12e511a241b24ca028ef834c698ecc068fde"} Sep 11 20:01:45.850884 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:45.850853 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b5b4c8886-wf7z8" event={"ID":"0d6de2e0-f1ba-443b-9624-7144ab50bf04","Type":"ContainerStarted","Data":"8443601012e0e9635848cd186e324f3d94c78d9e3b90791638294087b865dfbf"} Sep 11 20:01:45.850997 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:45.850927 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b5b4c8886-wf7z8" event={"ID":"0d6de2e0-f1ba-443b-9624-7144ab50bf04","Type":"ContainerStarted","Data":"dbbe43a4d843a8464cfccca08dfbfcc8107456b33bbd14057cac24b6e1b8dd81"} Sep 11 20:01:45.850997 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:45.850962 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b5b4c8886-wf7z8" event={"ID":"0d6de2e0-f1ba-443b-9624-7144ab50bf04","Type":"ContainerStarted","Data":"d00c546696d71a3e2c6733ea9efe0a9d5efb0aaac71c691f0627140acd63fac4"} Sep 11 20:01:46.860555 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:46.860512 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:46.861150 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:46.861128 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ddf54b53-5ff6-47d2-bfb1-260649d17ecb","Type":"ContainerStarted","Data":"390b8470cfd6002df93e9ca9cea59976dcae350c8a9538a91f5dd9bf3e675073"} Sep 11 20:01:46.864734 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:46.864709 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b5b4c8886-wf7z8" event={"ID":"0d6de2e0-f1ba-443b-9624-7144ab50bf04","Type":"ContainerStarted","Data":"36cd4b2e3af3baef385ab8651d13237a8f8831b67d3aef2f55ef8763fea0c92e"} Sep 11 20:01:46.864862 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:46.864779 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b5b4c8886-wf7z8" event={"ID":"0d6de2e0-f1ba-443b-9624-7144ab50bf04","Type":"ContainerStarted","Data":"29ee4131b29e2d6a21497f2497457e6d78269c5488e994abe672af715afa6ec2"} Sep 11 20:01:46.864862 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:46.864840 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b5b4c8886-wf7z8" event={"ID":"0d6de2e0-f1ba-443b-9624-7144ab50bf04","Type":"ContainerStarted","Data":"0269ea893297548bddaf724902f43eb5f8e94b009026351b259959b72347c46e"} Sep 11 20:01:46.864959 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:46.864883 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7b5b4c8886-wf7z8" Sep 11 20:01:46.889062 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:46.889007 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.995083147 podStartE2EDuration="9.888984406s" totalImagesPullingTime="5.873682292s" totalInitContainerRuntime="20.218967ms" isStatefulPod=false podCreationTimestamp="2026-09-11 20:01:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:01:46.886328755 +0000 UTC m=+76.934379098" watchObservedRunningTime="2026-09-11 20:01:46.888984406 +0000 UTC m=+76.937034754" Sep 11 20:01:46.908875 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:46.908814 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7b5b4c8886-wf7z8" podStartSLOduration=5.185382036 podStartE2EDuration="8.908793546s" totalImagesPullingTime="3.72341151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:01:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:01:46.90688646 +0000 UTC m=+76.954936829" watchObservedRunningTime="2026-09-11 20:01:46.908793546 +0000 UTC m=+76.956843893" Sep 11 20:01:46.966588 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:46.966542 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:47.030018 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:47.029986 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:48.878636 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:48.878596 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"71e7ab2f-2146-48a4-a9a0-b6ac3de11385","Type":"ContainerStarted","Data":"1e26caaa2e13b8294f8c63134fac5d98860699d9d18175113d22919b02b72e81"} Sep 11 20:01:48.878996 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:48.878663 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"71e7ab2f-2146-48a4-a9a0-b6ac3de11385","Type":"ContainerStarted","Data":"8be7262fa9233d57eaf437bb764aad9813b8595a143f36d393faaff8651e22a2"} Sep 11 20:01:48.878996 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:48.878689 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"71e7ab2f-2146-48a4-a9a0-b6ac3de11385","Type":"ContainerStarted","Data":"c6f7823a9b1c8f43ddc1bccbbe4e12f3bc445ae16a4f9836609282adb6b8b055"} Sep 11 20:01:48.878996 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:48.878721 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"71e7ab2f-2146-48a4-a9a0-b6ac3de11385","Type":"ContainerStarted","Data":"67da36248892982f4627891345c94bbbdd9f80fdbffd1df19b33417829189d75"} Sep 11 20:01:48.878996 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:48.878746 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"71e7ab2f-2146-48a4-a9a0-b6ac3de11385","Type":"ContainerStarted","Data":"850eec169eb8fb81640c4dbcef362bdd0da6665e52975d0df320f24d7584571b"} Sep 11 20:01:48.878996 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:48.878778 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"71e7ab2f-2146-48a4-a9a0-b6ac3de11385","Type":"ContainerStarted","Data":"4487c4d66faf952bccf5a2bd4d15a60ae982b3a15bcf7bfa7a0187d307cc19d2"} Sep 11 20:01:48.911904 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:48.911852 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=1.846922615 podStartE2EDuration="5.911833266s" totalImagesPullingTime="4.045808627s" totalInitContainerRuntime="19.102024ms" isStatefulPod=false podCreationTimestamp="2026-09-11 20:01:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:01:48.908982085 +0000 UTC m=+78.957032428" watchObservedRunningTime="2026-09-11 20:01:48.911833266 +0000 UTC m=+78.959883612" Sep 11 20:01:50.324545 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:50.324506 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:50.385415 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:50.385379 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:51.021835 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:51.021800 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:51.122402 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:51.122369 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:51.189687 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:51.189656 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:52.014317 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:52.014283 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-66875d4f5b-gt7sc" Sep 11 20:01:52.014732 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:52.014355 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-66875d4f5b-gt7sc" Sep 11 20:01:52.019043 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:52.019017 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-66875d4f5b-gt7sc" Sep 11 20:01:52.144001 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:52.143969 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:52.241807 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:52.241773 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:52.876953 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:52.876928 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7b5b4c8886-wf7z8" Sep 11 20:01:52.901820 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:52.901787 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-66875d4f5b-gt7sc" Sep 11 20:01:52.969017 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:52.968988 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f4bb78b9b-gwm4b"] Sep 11 20:01:53.074455 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:53.074358 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:53.176186 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:53.176099 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:53.276459 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:53.276413 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:53.391960 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:53.391930 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:01:53.871653 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:53.871602 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:53.940261 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:53.940231 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:54.107150 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:54.107115 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:54.211065 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:54.210989 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:59.099698 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:59.099652 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:59.192558 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:59.192528 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:59.397252 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:59.397182 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:59.466341 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:01:59.466312 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:02:13.840400 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:13.840366 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-wk9hk" Sep 11 20:02:14.065296 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:14.065265 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:02:14.124963 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:14.124881 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:02:17.987547 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:17.987511 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5f4bb78b9b-gwm4b" podUID="fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6" containerName="console" containerID="cri-o://0fd808914c55514783b98ea07beb410cda9c55abcafb05b2d497c7f3562ebb0f" gracePeriod=15 Sep 11 20:02:18.253979 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:18.253918 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f4bb78b9b-gwm4b" Sep 11 20:02:18.318432 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:18.318403 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6-console-config\") pod \"fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6\" (UID: \"fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6\") " Sep 11 20:02:18.318569 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:18.318462 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6-console-oauth-config\") pod \"fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6\" (UID: \"fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6\") " Sep 11 20:02:18.318569 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:18.318487 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6-oauth-serving-cert\") pod \"fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6\" (UID: \"fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6\") " Sep 11 20:02:18.318569 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:18.318502 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6-console-serving-cert\") pod \"fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6\" (UID: \"fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6\") " Sep 11 20:02:18.318569 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:18.318519 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6-service-ca\") pod \"fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6\" (UID: \"fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6\") " Sep 11 20:02:18.318569 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:18.318548 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6mbs\" (UniqueName: \"kubernetes.io/projected/fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6-kube-api-access-b6mbs\") pod \"fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6\" (UID: \"fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6\") " Sep 11 20:02:18.318893 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:18.318867 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6-console-config" (OuterVolumeSpecName: "console-config") pod "fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6" (UID: "fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:02:18.318954 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:18.318872 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6" (UID: "fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:02:18.318954 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:18.318924 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6-service-ca" (OuterVolumeSpecName: "service-ca") pod "fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6" (UID: "fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:02:18.320855 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:18.320827 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6-kube-api-access-b6mbs" (OuterVolumeSpecName: "kube-api-access-b6mbs") pod "fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6" (UID: "fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6"). InnerVolumeSpecName "kube-api-access-b6mbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:02:18.320953 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:18.320832 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6" (UID: "fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 20:02:18.320953 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:18.320896 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6" (UID: "fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 20:02:18.419975 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:18.419948 2546 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6-console-config\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:02:18.419975 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:18.419972 2546 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6-console-oauth-config\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:02:18.420106 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:18.419985 2546 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6-oauth-serving-cert\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:02:18.420106 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:18.419998 2546 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6-console-serving-cert\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:02:18.420106 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:18.420010 2546 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6-service-ca\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:02:18.420106 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:18.420021 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6mbs\" (UniqueName: \"kubernetes.io/projected/fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6-kube-api-access-b6mbs\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:02:18.997145 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:18.997103 2546 generic.go:356] "Generic (PLEG): container finished" podID="fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6" containerID="0fd808914c55514783b98ea07beb410cda9c55abcafb05b2d497c7f3562ebb0f" exitCode=0 Sep 11 20:02:18.997145 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:18.997145 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f4bb78b9b-gwm4b" Sep 11 20:02:18.997626 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:18.997188 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f4bb78b9b-gwm4b" event={"ID":"fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6","Type":"ContainerDied","Data":"0fd808914c55514783b98ea07beb410cda9c55abcafb05b2d497c7f3562ebb0f"} Sep 11 20:02:18.997626 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:18.997240 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f4bb78b9b-gwm4b" event={"ID":"fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6","Type":"ContainerDied","Data":"a4a516e1626f9806651a3bfe190f33195e609043939775df6390b2e21515af9b"} Sep 11 20:02:18.997626 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:18.997261 2546 scope.go:117] "RemoveContainer" containerID="0fd808914c55514783b98ea07beb410cda9c55abcafb05b2d497c7f3562ebb0f" Sep 11 20:02:19.005508 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:19.005491 2546 scope.go:117] "RemoveContainer" containerID="0fd808914c55514783b98ea07beb410cda9c55abcafb05b2d497c7f3562ebb0f" Sep 11 20:02:19.005804 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:02:19.005782 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fd808914c55514783b98ea07beb410cda9c55abcafb05b2d497c7f3562ebb0f\": container with ID starting with 0fd808914c55514783b98ea07beb410cda9c55abcafb05b2d497c7f3562ebb0f not found: ID does not exist" containerID="0fd808914c55514783b98ea07beb410cda9c55abcafb05b2d497c7f3562ebb0f" Sep 11 20:02:19.005864 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:19.005811 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fd808914c55514783b98ea07beb410cda9c55abcafb05b2d497c7f3562ebb0f"} err="failed to get container status \"0fd808914c55514783b98ea07beb410cda9c55abcafb05b2d497c7f3562ebb0f\": rpc error: code = NotFound desc = could not find container \"0fd808914c55514783b98ea07beb410cda9c55abcafb05b2d497c7f3562ebb0f\": container with ID starting with 0fd808914c55514783b98ea07beb410cda9c55abcafb05b2d497c7f3562ebb0f not found: ID does not exist" Sep 11 20:02:19.012083 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:19.012061 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f4bb78b9b-gwm4b"] Sep 11 20:02:19.017746 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:19.017726 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5f4bb78b9b-gwm4b"] Sep 11 20:02:20.447946 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:20.447911 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6" path="/var/lib/kubelet/pods/fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6/volumes" Sep 11 20:02:21.454856 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:21.454821 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:02:43.391977 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:43.391947 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:02:43.409981 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:43.409958 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:02:44.108468 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:44.108442 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:02:56.014642 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:56.014568 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:02:56.240799 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:56.240760 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:02:57.071141 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:57.071111 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 20:02:57.071623 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:57.071424 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ddf54b53-5ff6-47d2-bfb1-260649d17ecb" containerName="alertmanager" containerID="cri-o://3be599218d92c14d9973ffa3cd4f12e511a241b24ca028ef834c698ecc068fde" gracePeriod=120 Sep 11 20:02:57.071623 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:57.071462 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ddf54b53-5ff6-47d2-bfb1-260649d17ecb" containerName="kube-rbac-proxy-metric" containerID="cri-o://fa78a576841dfd3bf0754b37bf8dc2ba4fb8330c32decf03631ff15f4f4dc8bd" gracePeriod=120 Sep 11 20:02:57.071623 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:57.071553 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ddf54b53-5ff6-47d2-bfb1-260649d17ecb" containerName="kube-rbac-proxy-web" containerID="cri-o://47e36fec62fb15fc3ecbc383518bbead193e1cf4de3c5c2baeb44ddf70f93421" gracePeriod=120 Sep 11 20:02:57.071623 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:57.071597 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ddf54b53-5ff6-47d2-bfb1-260649d17ecb" containerName="prom-label-proxy" containerID="cri-o://390b8470cfd6002df93e9ca9cea59976dcae350c8a9538a91f5dd9bf3e675073" gracePeriod=120 Sep 11 20:02:57.071876 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:57.071607 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ddf54b53-5ff6-47d2-bfb1-260649d17ecb" containerName="kube-rbac-proxy" containerID="cri-o://af0c29c4fa1e3e932c8dbf144e67c942833eed221b6f79ef06edb06dbc453766" gracePeriod=120 Sep 11 20:02:57.071876 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:57.071607 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ddf54b53-5ff6-47d2-bfb1-260649d17ecb" containerName="config-reloader" containerID="cri-o://91d34e9fafd743d1cc62cc58f05b7da54aacdcfd87137e9483732b05c6934dd1" gracePeriod=120 Sep 11 20:02:58.148896 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:58.148861 2546 generic.go:356] "Generic (PLEG): container finished" podID="ddf54b53-5ff6-47d2-bfb1-260649d17ecb" containerID="390b8470cfd6002df93e9ca9cea59976dcae350c8a9538a91f5dd9bf3e675073" exitCode=0 Sep 11 20:02:58.148896 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:58.148888 2546 generic.go:356] "Generic (PLEG): container finished" podID="ddf54b53-5ff6-47d2-bfb1-260649d17ecb" containerID="af0c29c4fa1e3e932c8dbf144e67c942833eed221b6f79ef06edb06dbc453766" exitCode=0 Sep 11 20:02:58.148896 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:58.148894 2546 generic.go:356] "Generic (PLEG): container finished" podID="ddf54b53-5ff6-47d2-bfb1-260649d17ecb" containerID="91d34e9fafd743d1cc62cc58f05b7da54aacdcfd87137e9483732b05c6934dd1" exitCode=0 Sep 11 20:02:58.148896 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:58.148901 2546 generic.go:356] "Generic (PLEG): container finished" podID="ddf54b53-5ff6-47d2-bfb1-260649d17ecb" containerID="3be599218d92c14d9973ffa3cd4f12e511a241b24ca028ef834c698ecc068fde" exitCode=0 Sep 11 20:02:58.149412 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:58.148933 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ddf54b53-5ff6-47d2-bfb1-260649d17ecb","Type":"ContainerDied","Data":"390b8470cfd6002df93e9ca9cea59976dcae350c8a9538a91f5dd9bf3e675073"} Sep 11 20:02:58.149412 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:58.149016 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ddf54b53-5ff6-47d2-bfb1-260649d17ecb","Type":"ContainerDied","Data":"af0c29c4fa1e3e932c8dbf144e67c942833eed221b6f79ef06edb06dbc453766"} Sep 11 20:02:58.149412 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:58.149065 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ddf54b53-5ff6-47d2-bfb1-260649d17ecb","Type":"ContainerDied","Data":"91d34e9fafd743d1cc62cc58f05b7da54aacdcfd87137e9483732b05c6934dd1"} Sep 11 20:02:58.149412 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:58.149104 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ddf54b53-5ff6-47d2-bfb1-260649d17ecb","Type":"ContainerDied","Data":"3be599218d92c14d9973ffa3cd4f12e511a241b24ca028ef834c698ecc068fde"} Sep 11 20:02:58.411980 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:58.411932 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:02:58.507791 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:58.507753 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-config-volume\") pod \"ddf54b53-5ff6-47d2-bfb1-260649d17ecb\" (UID: \"ddf54b53-5ff6-47d2-bfb1-260649d17ecb\") " Sep 11 20:02:58.507791 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:58.507799 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-alertmanager-trusted-ca-bundle\") pod \"ddf54b53-5ff6-47d2-bfb1-260649d17ecb\" (UID: \"ddf54b53-5ff6-47d2-bfb1-260649d17ecb\") " Sep 11 20:02:58.508010 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:58.507821 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-cluster-tls-config\") pod \"ddf54b53-5ff6-47d2-bfb1-260649d17ecb\" (UID: \"ddf54b53-5ff6-47d2-bfb1-260649d17ecb\") " Sep 11 20:02:58.508010 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:58.507843 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-secret-alertmanager-kube-rbac-proxy-web\") pod \"ddf54b53-5ff6-47d2-bfb1-260649d17ecb\" (UID: \"ddf54b53-5ff6-47d2-bfb1-260649d17ecb\") " Sep 11 20:02:58.508010 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:58.507927 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-metrics-client-ca\") pod \"ddf54b53-5ff6-47d2-bfb1-260649d17ecb\" (UID: \"ddf54b53-5ff6-47d2-bfb1-260649d17ecb\") " Sep 11 20:02:58.508010 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:58.507976 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-config-out\") pod \"ddf54b53-5ff6-47d2-bfb1-260649d17ecb\" (UID: \"ddf54b53-5ff6-47d2-bfb1-260649d17ecb\") " Sep 11 20:02:58.508010 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:58.508002 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-alertmanager-main-db\") pod \"ddf54b53-5ff6-47d2-bfb1-260649d17ecb\" (UID: \"ddf54b53-5ff6-47d2-bfb1-260649d17ecb\") " Sep 11 20:02:58.508271 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:58.508034 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"ddf54b53-5ff6-47d2-bfb1-260649d17ecb\" (UID: \"ddf54b53-5ff6-47d2-bfb1-260649d17ecb\") " Sep 11 20:02:58.508271 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:58.508078 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f2g4s\" (UniqueName: \"kubernetes.io/projected/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-kube-api-access-f2g4s\") pod \"ddf54b53-5ff6-47d2-bfb1-260649d17ecb\" (UID: \"ddf54b53-5ff6-47d2-bfb1-260649d17ecb\") " Sep 11 20:02:58.508271 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:58.508102 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-web-config\") pod \"ddf54b53-5ff6-47d2-bfb1-260649d17ecb\" (UID: \"ddf54b53-5ff6-47d2-bfb1-260649d17ecb\") " Sep 11 20:02:58.508271 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:58.508135 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-secret-alertmanager-kube-rbac-proxy\") pod \"ddf54b53-5ff6-47d2-bfb1-260649d17ecb\" (UID: \"ddf54b53-5ff6-47d2-bfb1-260649d17ecb\") " Sep 11 20:02:58.508271 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:58.508189 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "ddf54b53-5ff6-47d2-bfb1-260649d17ecb" (UID: "ddf54b53-5ff6-47d2-bfb1-260649d17ecb"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:02:58.508271 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:58.508204 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-tls-assets\") pod \"ddf54b53-5ff6-47d2-bfb1-260649d17ecb\" (UID: \"ddf54b53-5ff6-47d2-bfb1-260649d17ecb\") " Sep 11 20:02:58.508271 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:58.508254 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-secret-alertmanager-main-tls\") pod \"ddf54b53-5ff6-47d2-bfb1-260649d17ecb\" (UID: \"ddf54b53-5ff6-47d2-bfb1-260649d17ecb\") " Sep 11 20:02:58.508271 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:58.508253 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "ddf54b53-5ff6-47d2-bfb1-260649d17ecb" (UID: "ddf54b53-5ff6-47d2-bfb1-260649d17ecb"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:02:58.508682 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:58.508521 2546 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-metrics-client-ca\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:02:58.508682 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:58.508541 2546 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:02:58.508789 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:58.508738 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "ddf54b53-5ff6-47d2-bfb1-260649d17ecb" (UID: "ddf54b53-5ff6-47d2-bfb1-260649d17ecb"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:02:58.511179 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:58.511135 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-config-volume" (OuterVolumeSpecName: "config-volume") pod "ddf54b53-5ff6-47d2-bfb1-260649d17ecb" (UID: "ddf54b53-5ff6-47d2-bfb1-260649d17ecb"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 20:02:58.511442 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:58.511417 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "ddf54b53-5ff6-47d2-bfb1-260649d17ecb" (UID: "ddf54b53-5ff6-47d2-bfb1-260649d17ecb"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 20:02:58.511645 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:58.511611 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-config-out" (OuterVolumeSpecName: "config-out") pod "ddf54b53-5ff6-47d2-bfb1-260649d17ecb" (UID: "ddf54b53-5ff6-47d2-bfb1-260649d17ecb"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:02:58.511645 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:58.511621 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "ddf54b53-5ff6-47d2-bfb1-260649d17ecb" (UID: "ddf54b53-5ff6-47d2-bfb1-260649d17ecb"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 20:02:58.511645 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:58.511632 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-kube-api-access-f2g4s" (OuterVolumeSpecName: "kube-api-access-f2g4s") pod "ddf54b53-5ff6-47d2-bfb1-260649d17ecb" (UID: "ddf54b53-5ff6-47d2-bfb1-260649d17ecb"). InnerVolumeSpecName "kube-api-access-f2g4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:02:58.511855 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:58.511757 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "ddf54b53-5ff6-47d2-bfb1-260649d17ecb" (UID: "ddf54b53-5ff6-47d2-bfb1-260649d17ecb"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 20:02:58.511910 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:58.511890 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "ddf54b53-5ff6-47d2-bfb1-260649d17ecb" (UID: "ddf54b53-5ff6-47d2-bfb1-260649d17ecb"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:02:58.512345 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:58.512320 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "ddf54b53-5ff6-47d2-bfb1-260649d17ecb" (UID: "ddf54b53-5ff6-47d2-bfb1-260649d17ecb"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 20:02:58.514842 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:58.514776 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "ddf54b53-5ff6-47d2-bfb1-260649d17ecb" (UID: "ddf54b53-5ff6-47d2-bfb1-260649d17ecb"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 20:02:58.521248 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:58.521227 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-web-config" (OuterVolumeSpecName: "web-config") pod "ddf54b53-5ff6-47d2-bfb1-260649d17ecb" (UID: "ddf54b53-5ff6-47d2-bfb1-260649d17ecb"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 20:02:58.609342 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:58.609316 2546 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-config-volume\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:02:58.609342 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:58.609337 2546 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-cluster-tls-config\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:02:58.609342 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:58.609347 2546 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:02:58.609513 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:58.609356 2546 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-config-out\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:02:58.609513 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:58.609365 2546 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-alertmanager-main-db\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:02:58.609513 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:58.609373 2546 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:02:58.609513 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:58.609383 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2g4s\" (UniqueName: \"kubernetes.io/projected/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-kube-api-access-f2g4s\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:02:58.609513 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:58.609391 2546 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-web-config\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:02:58.609513 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:58.609400 2546 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:02:58.609513 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:58.609409 2546 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-tls-assets\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:02:58.609513 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:58.609417 2546 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ddf54b53-5ff6-47d2-bfb1-260649d17ecb-secret-alertmanager-main-tls\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:02:59.155357 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.155329 2546 generic.go:356] "Generic (PLEG): container finished" podID="ddf54b53-5ff6-47d2-bfb1-260649d17ecb" containerID="fa78a576841dfd3bf0754b37bf8dc2ba4fb8330c32decf03631ff15f4f4dc8bd" exitCode=0 Sep 11 20:02:59.155357 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.155353 2546 generic.go:356] "Generic (PLEG): container finished" podID="ddf54b53-5ff6-47d2-bfb1-260649d17ecb" containerID="47e36fec62fb15fc3ecbc383518bbead193e1cf4de3c5c2baeb44ddf70f93421" exitCode=0 Sep 11 20:02:59.155847 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.155385 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ddf54b53-5ff6-47d2-bfb1-260649d17ecb","Type":"ContainerDied","Data":"fa78a576841dfd3bf0754b37bf8dc2ba4fb8330c32decf03631ff15f4f4dc8bd"} Sep 11 20:02:59.155847 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.155410 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:02:59.155847 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.155443 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ddf54b53-5ff6-47d2-bfb1-260649d17ecb","Type":"ContainerDied","Data":"47e36fec62fb15fc3ecbc383518bbead193e1cf4de3c5c2baeb44ddf70f93421"} Sep 11 20:02:59.155847 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.155471 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ddf54b53-5ff6-47d2-bfb1-260649d17ecb","Type":"ContainerDied","Data":"704accae14af3cbfe78f50bc27d4a331aed886998c7847cbb29cdb83d40076b0"} Sep 11 20:02:59.155847 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.155501 2546 scope.go:117] "RemoveContainer" containerID="390b8470cfd6002df93e9ca9cea59976dcae350c8a9538a91f5dd9bf3e675073" Sep 11 20:02:59.166733 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.166705 2546 scope.go:117] "RemoveContainer" containerID="fa78a576841dfd3bf0754b37bf8dc2ba4fb8330c32decf03631ff15f4f4dc8bd" Sep 11 20:02:59.173943 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.173923 2546 scope.go:117] "RemoveContainer" containerID="af0c29c4fa1e3e932c8dbf144e67c942833eed221b6f79ef06edb06dbc453766" Sep 11 20:02:59.180834 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.180810 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 20:02:59.181032 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.180976 2546 scope.go:117] "RemoveContainer" containerID="47e36fec62fb15fc3ecbc383518bbead193e1cf4de3c5c2baeb44ddf70f93421" Sep 11 20:02:59.183968 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.183946 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 20:02:59.187878 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.187720 2546 scope.go:117] "RemoveContainer" containerID="91d34e9fafd743d1cc62cc58f05b7da54aacdcfd87137e9483732b05c6934dd1" Sep 11 20:02:59.193862 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.193846 2546 scope.go:117] "RemoveContainer" containerID="3be599218d92c14d9973ffa3cd4f12e511a241b24ca028ef834c698ecc068fde" Sep 11 20:02:59.200191 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.200173 2546 scope.go:117] "RemoveContainer" containerID="0af2d149007812e42ac45dca18739f71b1642eeac5cffc77d81c37da836723ae" Sep 11 20:02:59.206786 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.206771 2546 scope.go:117] "RemoveContainer" containerID="390b8470cfd6002df93e9ca9cea59976dcae350c8a9538a91f5dd9bf3e675073" Sep 11 20:02:59.207024 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:02:59.207008 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"390b8470cfd6002df93e9ca9cea59976dcae350c8a9538a91f5dd9bf3e675073\": container with ID starting with 390b8470cfd6002df93e9ca9cea59976dcae350c8a9538a91f5dd9bf3e675073 not found: ID does not exist" containerID="390b8470cfd6002df93e9ca9cea59976dcae350c8a9538a91f5dd9bf3e675073" Sep 11 20:02:59.207075 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.207030 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"390b8470cfd6002df93e9ca9cea59976dcae350c8a9538a91f5dd9bf3e675073"} err="failed to get container status \"390b8470cfd6002df93e9ca9cea59976dcae350c8a9538a91f5dd9bf3e675073\": rpc error: code = NotFound desc = could not find container \"390b8470cfd6002df93e9ca9cea59976dcae350c8a9538a91f5dd9bf3e675073\": container with ID starting with 390b8470cfd6002df93e9ca9cea59976dcae350c8a9538a91f5dd9bf3e675073 not found: ID does not exist" Sep 11 20:02:59.207075 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.207045 2546 scope.go:117] "RemoveContainer" containerID="fa78a576841dfd3bf0754b37bf8dc2ba4fb8330c32decf03631ff15f4f4dc8bd" Sep 11 20:02:59.207286 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:02:59.207268 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa78a576841dfd3bf0754b37bf8dc2ba4fb8330c32decf03631ff15f4f4dc8bd\": container with ID starting with fa78a576841dfd3bf0754b37bf8dc2ba4fb8330c32decf03631ff15f4f4dc8bd not found: ID does not exist" containerID="fa78a576841dfd3bf0754b37bf8dc2ba4fb8330c32decf03631ff15f4f4dc8bd" Sep 11 20:02:59.207321 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.207300 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa78a576841dfd3bf0754b37bf8dc2ba4fb8330c32decf03631ff15f4f4dc8bd"} err="failed to get container status \"fa78a576841dfd3bf0754b37bf8dc2ba4fb8330c32decf03631ff15f4f4dc8bd\": rpc error: code = NotFound desc = could not find container \"fa78a576841dfd3bf0754b37bf8dc2ba4fb8330c32decf03631ff15f4f4dc8bd\": container with ID starting with fa78a576841dfd3bf0754b37bf8dc2ba4fb8330c32decf03631ff15f4f4dc8bd not found: ID does not exist" Sep 11 20:02:59.207321 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.207315 2546 scope.go:117] "RemoveContainer" containerID="af0c29c4fa1e3e932c8dbf144e67c942833eed221b6f79ef06edb06dbc453766" Sep 11 20:02:59.207548 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:02:59.207531 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af0c29c4fa1e3e932c8dbf144e67c942833eed221b6f79ef06edb06dbc453766\": container with ID starting with af0c29c4fa1e3e932c8dbf144e67c942833eed221b6f79ef06edb06dbc453766 not found: ID does not exist" containerID="af0c29c4fa1e3e932c8dbf144e67c942833eed221b6f79ef06edb06dbc453766" Sep 11 20:02:59.207618 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.207551 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af0c29c4fa1e3e932c8dbf144e67c942833eed221b6f79ef06edb06dbc453766"} err="failed to get container status \"af0c29c4fa1e3e932c8dbf144e67c942833eed221b6f79ef06edb06dbc453766\": rpc error: code = NotFound desc = could not find container \"af0c29c4fa1e3e932c8dbf144e67c942833eed221b6f79ef06edb06dbc453766\": container with ID starting with af0c29c4fa1e3e932c8dbf144e67c942833eed221b6f79ef06edb06dbc453766 not found: ID does not exist" Sep 11 20:02:59.207618 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.207568 2546 scope.go:117] "RemoveContainer" containerID="47e36fec62fb15fc3ecbc383518bbead193e1cf4de3c5c2baeb44ddf70f93421" Sep 11 20:02:59.207841 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:02:59.207826 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47e36fec62fb15fc3ecbc383518bbead193e1cf4de3c5c2baeb44ddf70f93421\": container with ID starting with 47e36fec62fb15fc3ecbc383518bbead193e1cf4de3c5c2baeb44ddf70f93421 not found: ID does not exist" containerID="47e36fec62fb15fc3ecbc383518bbead193e1cf4de3c5c2baeb44ddf70f93421" Sep 11 20:02:59.207889 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.207844 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47e36fec62fb15fc3ecbc383518bbead193e1cf4de3c5c2baeb44ddf70f93421"} err="failed to get container status \"47e36fec62fb15fc3ecbc383518bbead193e1cf4de3c5c2baeb44ddf70f93421\": rpc error: code = NotFound desc = could not find container \"47e36fec62fb15fc3ecbc383518bbead193e1cf4de3c5c2baeb44ddf70f93421\": container with ID starting with 47e36fec62fb15fc3ecbc383518bbead193e1cf4de3c5c2baeb44ddf70f93421 not found: ID does not exist" Sep 11 20:02:59.207889 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.207863 2546 scope.go:117] "RemoveContainer" containerID="91d34e9fafd743d1cc62cc58f05b7da54aacdcfd87137e9483732b05c6934dd1" Sep 11 20:02:59.208067 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:02:59.208053 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91d34e9fafd743d1cc62cc58f05b7da54aacdcfd87137e9483732b05c6934dd1\": container with ID starting with 91d34e9fafd743d1cc62cc58f05b7da54aacdcfd87137e9483732b05c6934dd1 not found: ID does not exist" containerID="91d34e9fafd743d1cc62cc58f05b7da54aacdcfd87137e9483732b05c6934dd1" Sep 11 20:02:59.208106 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.208070 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91d34e9fafd743d1cc62cc58f05b7da54aacdcfd87137e9483732b05c6934dd1"} err="failed to get container status \"91d34e9fafd743d1cc62cc58f05b7da54aacdcfd87137e9483732b05c6934dd1\": rpc error: code = NotFound desc = could not find container \"91d34e9fafd743d1cc62cc58f05b7da54aacdcfd87137e9483732b05c6934dd1\": container with ID starting with 91d34e9fafd743d1cc62cc58f05b7da54aacdcfd87137e9483732b05c6934dd1 not found: ID does not exist" Sep 11 20:02:59.208106 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.208081 2546 scope.go:117] "RemoveContainer" containerID="3be599218d92c14d9973ffa3cd4f12e511a241b24ca028ef834c698ecc068fde" Sep 11 20:02:59.208301 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:02:59.208286 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3be599218d92c14d9973ffa3cd4f12e511a241b24ca028ef834c698ecc068fde\": container with ID starting with 3be599218d92c14d9973ffa3cd4f12e511a241b24ca028ef834c698ecc068fde not found: ID does not exist" containerID="3be599218d92c14d9973ffa3cd4f12e511a241b24ca028ef834c698ecc068fde" Sep 11 20:02:59.208339 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.208304 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3be599218d92c14d9973ffa3cd4f12e511a241b24ca028ef834c698ecc068fde"} err="failed to get container status \"3be599218d92c14d9973ffa3cd4f12e511a241b24ca028ef834c698ecc068fde\": rpc error: code = NotFound desc = could not find container \"3be599218d92c14d9973ffa3cd4f12e511a241b24ca028ef834c698ecc068fde\": container with ID starting with 3be599218d92c14d9973ffa3cd4f12e511a241b24ca028ef834c698ecc068fde not found: ID does not exist" Sep 11 20:02:59.208339 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.208317 2546 scope.go:117] "RemoveContainer" containerID="0af2d149007812e42ac45dca18739f71b1642eeac5cffc77d81c37da836723ae" Sep 11 20:02:59.208498 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:02:59.208484 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0af2d149007812e42ac45dca18739f71b1642eeac5cffc77d81c37da836723ae\": container with ID starting with 0af2d149007812e42ac45dca18739f71b1642eeac5cffc77d81c37da836723ae not found: ID does not exist" containerID="0af2d149007812e42ac45dca18739f71b1642eeac5cffc77d81c37da836723ae" Sep 11 20:02:59.208532 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.208500 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0af2d149007812e42ac45dca18739f71b1642eeac5cffc77d81c37da836723ae"} err="failed to get container status \"0af2d149007812e42ac45dca18739f71b1642eeac5cffc77d81c37da836723ae\": rpc error: code = NotFound desc = could not find container \"0af2d149007812e42ac45dca18739f71b1642eeac5cffc77d81c37da836723ae\": container with ID starting with 0af2d149007812e42ac45dca18739f71b1642eeac5cffc77d81c37da836723ae not found: ID does not exist" Sep 11 20:02:59.208532 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.208512 2546 scope.go:117] "RemoveContainer" containerID="390b8470cfd6002df93e9ca9cea59976dcae350c8a9538a91f5dd9bf3e675073" Sep 11 20:02:59.208754 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.208737 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"390b8470cfd6002df93e9ca9cea59976dcae350c8a9538a91f5dd9bf3e675073"} err="failed to get container status \"390b8470cfd6002df93e9ca9cea59976dcae350c8a9538a91f5dd9bf3e675073\": rpc error: code = NotFound desc = could not find container \"390b8470cfd6002df93e9ca9cea59976dcae350c8a9538a91f5dd9bf3e675073\": container with ID starting with 390b8470cfd6002df93e9ca9cea59976dcae350c8a9538a91f5dd9bf3e675073 not found: ID does not exist" Sep 11 20:02:59.208809 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.208755 2546 scope.go:117] "RemoveContainer" containerID="fa78a576841dfd3bf0754b37bf8dc2ba4fb8330c32decf03631ff15f4f4dc8bd" Sep 11 20:02:59.208956 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.208941 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa78a576841dfd3bf0754b37bf8dc2ba4fb8330c32decf03631ff15f4f4dc8bd"} err="failed to get container status \"fa78a576841dfd3bf0754b37bf8dc2ba4fb8330c32decf03631ff15f4f4dc8bd\": rpc error: code = NotFound desc = could not find container \"fa78a576841dfd3bf0754b37bf8dc2ba4fb8330c32decf03631ff15f4f4dc8bd\": container with ID starting with fa78a576841dfd3bf0754b37bf8dc2ba4fb8330c32decf03631ff15f4f4dc8bd not found: ID does not exist" Sep 11 20:02:59.208997 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.208957 2546 scope.go:117] "RemoveContainer" containerID="af0c29c4fa1e3e932c8dbf144e67c942833eed221b6f79ef06edb06dbc453766" Sep 11 20:02:59.209175 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.209158 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af0c29c4fa1e3e932c8dbf144e67c942833eed221b6f79ef06edb06dbc453766"} err="failed to get container status \"af0c29c4fa1e3e932c8dbf144e67c942833eed221b6f79ef06edb06dbc453766\": rpc error: code = NotFound desc = could not find container \"af0c29c4fa1e3e932c8dbf144e67c942833eed221b6f79ef06edb06dbc453766\": container with ID starting with af0c29c4fa1e3e932c8dbf144e67c942833eed221b6f79ef06edb06dbc453766 not found: ID does not exist" Sep 11 20:02:59.209175 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.209174 2546 scope.go:117] "RemoveContainer" containerID="47e36fec62fb15fc3ecbc383518bbead193e1cf4de3c5c2baeb44ddf70f93421" Sep 11 20:02:59.209372 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.209355 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47e36fec62fb15fc3ecbc383518bbead193e1cf4de3c5c2baeb44ddf70f93421"} err="failed to get container status \"47e36fec62fb15fc3ecbc383518bbead193e1cf4de3c5c2baeb44ddf70f93421\": rpc error: code = NotFound desc = could not find container \"47e36fec62fb15fc3ecbc383518bbead193e1cf4de3c5c2baeb44ddf70f93421\": container with ID starting with 47e36fec62fb15fc3ecbc383518bbead193e1cf4de3c5c2baeb44ddf70f93421 not found: ID does not exist" Sep 11 20:02:59.209415 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.209372 2546 scope.go:117] "RemoveContainer" containerID="91d34e9fafd743d1cc62cc58f05b7da54aacdcfd87137e9483732b05c6934dd1" Sep 11 20:02:59.209634 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.209617 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91d34e9fafd743d1cc62cc58f05b7da54aacdcfd87137e9483732b05c6934dd1"} err="failed to get container status \"91d34e9fafd743d1cc62cc58f05b7da54aacdcfd87137e9483732b05c6934dd1\": rpc error: code = NotFound desc = could not find container \"91d34e9fafd743d1cc62cc58f05b7da54aacdcfd87137e9483732b05c6934dd1\": container with ID starting with 91d34e9fafd743d1cc62cc58f05b7da54aacdcfd87137e9483732b05c6934dd1 not found: ID does not exist" Sep 11 20:02:59.209634 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.209633 2546 scope.go:117] "RemoveContainer" containerID="3be599218d92c14d9973ffa3cd4f12e511a241b24ca028ef834c698ecc068fde" Sep 11 20:02:59.209831 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.209817 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3be599218d92c14d9973ffa3cd4f12e511a241b24ca028ef834c698ecc068fde"} err="failed to get container status \"3be599218d92c14d9973ffa3cd4f12e511a241b24ca028ef834c698ecc068fde\": rpc error: code = NotFound desc = could not find container \"3be599218d92c14d9973ffa3cd4f12e511a241b24ca028ef834c698ecc068fde\": container with ID starting with 3be599218d92c14d9973ffa3cd4f12e511a241b24ca028ef834c698ecc068fde not found: ID does not exist" Sep 11 20:02:59.209871 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.209831 2546 scope.go:117] "RemoveContainer" containerID="0af2d149007812e42ac45dca18739f71b1642eeac5cffc77d81c37da836723ae" Sep 11 20:02:59.210043 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.210027 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0af2d149007812e42ac45dca18739f71b1642eeac5cffc77d81c37da836723ae"} err="failed to get container status \"0af2d149007812e42ac45dca18739f71b1642eeac5cffc77d81c37da836723ae\": rpc error: code = NotFound desc = could not find container \"0af2d149007812e42ac45dca18739f71b1642eeac5cffc77d81c37da836723ae\": container with ID starting with 0af2d149007812e42ac45dca18739f71b1642eeac5cffc77d81c37da836723ae not found: ID does not exist" Sep 11 20:02:59.214273 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.214253 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 20:02:59.214523 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.214509 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddf54b53-5ff6-47d2-bfb1-260649d17ecb" containerName="prom-label-proxy" Sep 11 20:02:59.214523 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.214524 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddf54b53-5ff6-47d2-bfb1-260649d17ecb" containerName="prom-label-proxy" Sep 11 20:02:59.214662 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.214533 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddf54b53-5ff6-47d2-bfb1-260649d17ecb" containerName="alertmanager" Sep 11 20:02:59.214662 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.214539 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddf54b53-5ff6-47d2-bfb1-260649d17ecb" containerName="alertmanager" Sep 11 20:02:59.214662 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.214546 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6" containerName="console" Sep 11 20:02:59.214662 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.214551 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6" containerName="console" Sep 11 20:02:59.214662 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.214558 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddf54b53-5ff6-47d2-bfb1-260649d17ecb" containerName="kube-rbac-proxy-metric" Sep 11 20:02:59.214662 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.214563 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddf54b53-5ff6-47d2-bfb1-260649d17ecb" containerName="kube-rbac-proxy-metric" Sep 11 20:02:59.214662 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.214587 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddf54b53-5ff6-47d2-bfb1-260649d17ecb" containerName="kube-rbac-proxy-web" Sep 11 20:02:59.214662 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.214596 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddf54b53-5ff6-47d2-bfb1-260649d17ecb" containerName="kube-rbac-proxy-web" Sep 11 20:02:59.214662 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.214604 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddf54b53-5ff6-47d2-bfb1-260649d17ecb" containerName="kube-rbac-proxy" Sep 11 20:02:59.214662 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.214609 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddf54b53-5ff6-47d2-bfb1-260649d17ecb" containerName="kube-rbac-proxy" Sep 11 20:02:59.214662 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.214617 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddf54b53-5ff6-47d2-bfb1-260649d17ecb" containerName="init-config-reloader" Sep 11 20:02:59.214662 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.214621 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddf54b53-5ff6-47d2-bfb1-260649d17ecb" containerName="init-config-reloader" Sep 11 20:02:59.214662 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.214632 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddf54b53-5ff6-47d2-bfb1-260649d17ecb" containerName="config-reloader" Sep 11 20:02:59.214662 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.214637 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddf54b53-5ff6-47d2-bfb1-260649d17ecb" containerName="config-reloader" Sep 11 20:02:59.215093 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.214680 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddf54b53-5ff6-47d2-bfb1-260649d17ecb" containerName="kube-rbac-proxy-web" Sep 11 20:02:59.215093 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.214689 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddf54b53-5ff6-47d2-bfb1-260649d17ecb" containerName="prom-label-proxy" Sep 11 20:02:59.215093 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.214697 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddf54b53-5ff6-47d2-bfb1-260649d17ecb" containerName="config-reloader" Sep 11 20:02:59.215093 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.214706 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddf54b53-5ff6-47d2-bfb1-260649d17ecb" containerName="alertmanager" Sep 11 20:02:59.215093 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.214713 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddf54b53-5ff6-47d2-bfb1-260649d17ecb" containerName="kube-rbac-proxy" Sep 11 20:02:59.215093 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.214720 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddf54b53-5ff6-47d2-bfb1-260649d17ecb" containerName="kube-rbac-proxy-metric" Sep 11 20:02:59.215093 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.214728 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbc2ea33-f0f9-4b0c-b1ae-b36f6cdae4e6" containerName="console" Sep 11 20:02:59.219464 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.219449 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:02:59.221458 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.221439 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 11 20:02:59.221620 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.221607 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 11 20:02:59.221703 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.221611 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 11 20:02:59.221703 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.221676 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 11 20:02:59.221970 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.221927 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 11 20:02:59.222072 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.221977 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 11 20:02:59.222072 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.221981 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 11 20:02:59.222072 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.221983 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-p4rwx\"" Sep 11 20:02:59.224384 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.222557 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 11 20:02:59.230253 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.230234 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 11 20:02:59.233724 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.233703 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 20:02:59.313423 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.313397 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ebf73f5-6a83-4e74-ab9a-28e881e2c57d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3ebf73f5-6a83-4e74-ab9a-28e881e2c57d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:02:59.313522 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.313431 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3ebf73f5-6a83-4e74-ab9a-28e881e2c57d-config-volume\") pod \"alertmanager-main-0\" (UID: \"3ebf73f5-6a83-4e74-ab9a-28e881e2c57d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:02:59.313522 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.313451 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3ebf73f5-6a83-4e74-ab9a-28e881e2c57d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3ebf73f5-6a83-4e74-ab9a-28e881e2c57d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:02:59.313607 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.313512 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3ebf73f5-6a83-4e74-ab9a-28e881e2c57d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3ebf73f5-6a83-4e74-ab9a-28e881e2c57d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:02:59.313607 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.313559 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3ebf73f5-6a83-4e74-ab9a-28e881e2c57d-web-config\") pod \"alertmanager-main-0\" (UID: \"3ebf73f5-6a83-4e74-ab9a-28e881e2c57d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:02:59.313673 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.313619 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3ebf73f5-6a83-4e74-ab9a-28e881e2c57d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3ebf73f5-6a83-4e74-ab9a-28e881e2c57d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:02:59.313673 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.313638 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3ebf73f5-6a83-4e74-ab9a-28e881e2c57d-config-out\") pod \"alertmanager-main-0\" (UID: \"3ebf73f5-6a83-4e74-ab9a-28e881e2c57d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:02:59.313673 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.313654 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ebf73f5-6a83-4e74-ab9a-28e881e2c57d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3ebf73f5-6a83-4e74-ab9a-28e881e2c57d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:02:59.313762 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.313706 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl4v4\" (UniqueName: \"kubernetes.io/projected/3ebf73f5-6a83-4e74-ab9a-28e881e2c57d-kube-api-access-nl4v4\") pod \"alertmanager-main-0\" (UID: \"3ebf73f5-6a83-4e74-ab9a-28e881e2c57d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:02:59.313762 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.313739 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3ebf73f5-6a83-4e74-ab9a-28e881e2c57d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3ebf73f5-6a83-4e74-ab9a-28e881e2c57d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:02:59.313828 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.313759 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ebf73f5-6a83-4e74-ab9a-28e881e2c57d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3ebf73f5-6a83-4e74-ab9a-28e881e2c57d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:02:59.313828 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.313780 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ebf73f5-6a83-4e74-ab9a-28e881e2c57d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3ebf73f5-6a83-4e74-ab9a-28e881e2c57d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:02:59.313828 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.313799 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3ebf73f5-6a83-4e74-ab9a-28e881e2c57d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3ebf73f5-6a83-4e74-ab9a-28e881e2c57d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:02:59.414522 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.414458 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3ebf73f5-6a83-4e74-ab9a-28e881e2c57d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3ebf73f5-6a83-4e74-ab9a-28e881e2c57d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:02:59.414522 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.414488 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3ebf73f5-6a83-4e74-ab9a-28e881e2c57d-config-out\") pod \"alertmanager-main-0\" (UID: \"3ebf73f5-6a83-4e74-ab9a-28e881e2c57d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:02:59.414522 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.414505 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ebf73f5-6a83-4e74-ab9a-28e881e2c57d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3ebf73f5-6a83-4e74-ab9a-28e881e2c57d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:02:59.414752 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.414526 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl4v4\" (UniqueName: \"kubernetes.io/projected/3ebf73f5-6a83-4e74-ab9a-28e881e2c57d-kube-api-access-nl4v4\") pod \"alertmanager-main-0\" (UID: \"3ebf73f5-6a83-4e74-ab9a-28e881e2c57d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:02:59.414752 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.414551 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3ebf73f5-6a83-4e74-ab9a-28e881e2c57d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3ebf73f5-6a83-4e74-ab9a-28e881e2c57d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:02:59.414752 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.414592 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ebf73f5-6a83-4e74-ab9a-28e881e2c57d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3ebf73f5-6a83-4e74-ab9a-28e881e2c57d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:02:59.414752 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.414628 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ebf73f5-6a83-4e74-ab9a-28e881e2c57d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3ebf73f5-6a83-4e74-ab9a-28e881e2c57d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:02:59.414752 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.414658 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3ebf73f5-6a83-4e74-ab9a-28e881e2c57d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3ebf73f5-6a83-4e74-ab9a-28e881e2c57d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:02:59.414752 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.414684 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ebf73f5-6a83-4e74-ab9a-28e881e2c57d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3ebf73f5-6a83-4e74-ab9a-28e881e2c57d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:02:59.414752 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.414717 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3ebf73f5-6a83-4e74-ab9a-28e881e2c57d-config-volume\") pod \"alertmanager-main-0\" (UID: \"3ebf73f5-6a83-4e74-ab9a-28e881e2c57d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:02:59.414752 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.414745 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3ebf73f5-6a83-4e74-ab9a-28e881e2c57d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3ebf73f5-6a83-4e74-ab9a-28e881e2c57d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:02:59.415126 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.414779 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3ebf73f5-6a83-4e74-ab9a-28e881e2c57d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3ebf73f5-6a83-4e74-ab9a-28e881e2c57d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:02:59.415126 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.414811 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3ebf73f5-6a83-4e74-ab9a-28e881e2c57d-web-config\") pod \"alertmanager-main-0\" (UID: \"3ebf73f5-6a83-4e74-ab9a-28e881e2c57d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:02:59.415126 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.414914 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3ebf73f5-6a83-4e74-ab9a-28e881e2c57d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3ebf73f5-6a83-4e74-ab9a-28e881e2c57d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:02:59.415800 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.415476 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ebf73f5-6a83-4e74-ab9a-28e881e2c57d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3ebf73f5-6a83-4e74-ab9a-28e881e2c57d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:02:59.415800 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.415550 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ebf73f5-6a83-4e74-ab9a-28e881e2c57d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3ebf73f5-6a83-4e74-ab9a-28e881e2c57d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:02:59.417385 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.417354 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3ebf73f5-6a83-4e74-ab9a-28e881e2c57d-config-out\") pod \"alertmanager-main-0\" (UID: \"3ebf73f5-6a83-4e74-ab9a-28e881e2c57d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:02:59.417945 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.417519 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3ebf73f5-6a83-4e74-ab9a-28e881e2c57d-config-volume\") pod \"alertmanager-main-0\" (UID: \"3ebf73f5-6a83-4e74-ab9a-28e881e2c57d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:02:59.417945 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.417809 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3ebf73f5-6a83-4e74-ab9a-28e881e2c57d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3ebf73f5-6a83-4e74-ab9a-28e881e2c57d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:02:59.417945 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.417870 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ebf73f5-6a83-4e74-ab9a-28e881e2c57d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3ebf73f5-6a83-4e74-ab9a-28e881e2c57d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:02:59.417945 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.417880 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3ebf73f5-6a83-4e74-ab9a-28e881e2c57d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3ebf73f5-6a83-4e74-ab9a-28e881e2c57d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:02:59.417945 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.417883 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3ebf73f5-6a83-4e74-ab9a-28e881e2c57d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3ebf73f5-6a83-4e74-ab9a-28e881e2c57d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:02:59.418251 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.418171 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ebf73f5-6a83-4e74-ab9a-28e881e2c57d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3ebf73f5-6a83-4e74-ab9a-28e881e2c57d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:02:59.418988 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.418965 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3ebf73f5-6a83-4e74-ab9a-28e881e2c57d-web-config\") pod \"alertmanager-main-0\" (UID: \"3ebf73f5-6a83-4e74-ab9a-28e881e2c57d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:02:59.419308 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.419285 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3ebf73f5-6a83-4e74-ab9a-28e881e2c57d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3ebf73f5-6a83-4e74-ab9a-28e881e2c57d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:02:59.422182 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.422161 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl4v4\" (UniqueName: \"kubernetes.io/projected/3ebf73f5-6a83-4e74-ab9a-28e881e2c57d-kube-api-access-nl4v4\") pod \"alertmanager-main-0\" (UID: \"3ebf73f5-6a83-4e74-ab9a-28e881e2c57d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:02:59.534159 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.534139 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 11 20:02:59.814269 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:02:59.814169 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 20:02:59.817604 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:02:59.817542 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ebf73f5_6a83_4e74_ab9a_28e881e2c57d.slice/crio-97fbf34a8365154e8f124a99ba428fabb9101bc4021a41e6a55ed5d7f5bbea14 WatchSource:0}: Error finding container 97fbf34a8365154e8f124a99ba428fabb9101bc4021a41e6a55ed5d7f5bbea14: Status 404 returned error can't find the container with id 97fbf34a8365154e8f124a99ba428fabb9101bc4021a41e6a55ed5d7f5bbea14 Sep 11 20:03:00.160439 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:00.160405 2546 generic.go:356] "Generic (PLEG): container finished" podID="3ebf73f5-6a83-4e74-ab9a-28e881e2c57d" containerID="87130f74b40f8c9f3ff02dab0ee92502b485bbaa75c57813062959cf6e21fe0e" exitCode=0 Sep 11 20:03:00.160845 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:00.160495 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ebf73f5-6a83-4e74-ab9a-28e881e2c57d","Type":"ContainerDied","Data":"87130f74b40f8c9f3ff02dab0ee92502b485bbaa75c57813062959cf6e21fe0e"} Sep 11 20:03:00.160845 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:00.160564 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ebf73f5-6a83-4e74-ab9a-28e881e2c57d","Type":"ContainerStarted","Data":"97fbf34a8365154e8f124a99ba428fabb9101bc4021a41e6a55ed5d7f5bbea14"} Sep 11 20:03:00.449366 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:00.449336 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddf54b53-5ff6-47d2-bfb1-260649d17ecb" path="/var/lib/kubelet/pods/ddf54b53-5ff6-47d2-bfb1-260649d17ecb/volumes" Sep 11 20:03:01.088853 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.088812 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-fc5d4f5c9-bksvl"] Sep 11 20:03:01.092807 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.092785 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-fc5d4f5c9-bksvl" Sep 11 20:03:01.094985 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.094954 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 11 20:03:01.094985 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.094978 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-k9ms5\"" Sep 11 20:03:01.095160 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.094961 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 11 20:03:01.095160 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.095058 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 11 20:03:01.095160 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.095126 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 11 20:03:01.095387 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.095373 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 11 20:03:01.100193 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.100169 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 11 20:03:01.104731 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.104711 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-fc5d4f5c9-bksvl"] Sep 11 20:03:01.169272 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.169242 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ebf73f5-6a83-4e74-ab9a-28e881e2c57d","Type":"ContainerStarted","Data":"6511f0a943ebdd8cd04396d324535abb8138ffd5f80b25f64ecee822fbf570da"} Sep 11 20:03:01.169605 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.169303 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ebf73f5-6a83-4e74-ab9a-28e881e2c57d","Type":"ContainerStarted","Data":"288822b92bbbf119c70d6478c551ee1dfb8eaa03285245f0189f4897bcbebbb8"} Sep 11 20:03:01.169605 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.169329 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ebf73f5-6a83-4e74-ab9a-28e881e2c57d","Type":"ContainerStarted","Data":"a62e0e02881ce1277f85bf988be5631af7d86f82dad29b6dff2ca3a97edc20ea"} Sep 11 20:03:01.169605 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.169357 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ebf73f5-6a83-4e74-ab9a-28e881e2c57d","Type":"ContainerStarted","Data":"c7d6cd83109148c70d06d43bbb47072005bb71178b9d1733991c30723ebc6bda"} Sep 11 20:03:01.169605 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.169380 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ebf73f5-6a83-4e74-ab9a-28e881e2c57d","Type":"ContainerStarted","Data":"04d260ffaf8ef5b14a3dcd6ff2fd45bd0251899edb2231a2a9ba4c4233b3441d"} Sep 11 20:03:01.169605 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.169402 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ebf73f5-6a83-4e74-ab9a-28e881e2c57d","Type":"ContainerStarted","Data":"21dc07f5209f3cd89060e2d3e6a284104f1bf9793c1dad4032d2645812969ae8"} Sep 11 20:03:01.193978 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.193918 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.179387124 podStartE2EDuration="2.193897336s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.510212ms" isStatefulPod=false podCreationTimestamp="2026-09-11 20:02:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:03:01.191911312 +0000 UTC m=+151.239961689" watchObservedRunningTime="2026-09-11 20:03:01.193897336 +0000 UTC m=+151.241947679" Sep 11 20:03:01.228095 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.228064 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9849d8f-4428-4c98-9aeb-f02cc00775d3-serving-certs-ca-bundle\") pod \"telemeter-client-fc5d4f5c9-bksvl\" (UID: \"e9849d8f-4428-4c98-9aeb-f02cc00775d3\") " pod="openshift-monitoring/telemeter-client-fc5d4f5c9-bksvl" Sep 11 20:03:01.228257 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.228119 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn24p\" (UniqueName: \"kubernetes.io/projected/e9849d8f-4428-4c98-9aeb-f02cc00775d3-kube-api-access-rn24p\") pod \"telemeter-client-fc5d4f5c9-bksvl\" (UID: \"e9849d8f-4428-4c98-9aeb-f02cc00775d3\") " pod="openshift-monitoring/telemeter-client-fc5d4f5c9-bksvl" Sep 11 20:03:01.228257 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.228147 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e9849d8f-4428-4c98-9aeb-f02cc00775d3-telemeter-client-tls\") pod \"telemeter-client-fc5d4f5c9-bksvl\" (UID: \"e9849d8f-4428-4c98-9aeb-f02cc00775d3\") " pod="openshift-monitoring/telemeter-client-fc5d4f5c9-bksvl" Sep 11 20:03:01.228381 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.228318 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9849d8f-4428-4c98-9aeb-f02cc00775d3-telemeter-trusted-ca-bundle\") pod \"telemeter-client-fc5d4f5c9-bksvl\" (UID: \"e9849d8f-4428-4c98-9aeb-f02cc00775d3\") " pod="openshift-monitoring/telemeter-client-fc5d4f5c9-bksvl" Sep 11 20:03:01.228536 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.228512 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e9849d8f-4428-4c98-9aeb-f02cc00775d3-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-fc5d4f5c9-bksvl\" (UID: \"e9849d8f-4428-4c98-9aeb-f02cc00775d3\") " pod="openshift-monitoring/telemeter-client-fc5d4f5c9-bksvl" Sep 11 20:03:01.228632 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.228586 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e9849d8f-4428-4c98-9aeb-f02cc00775d3-secret-telemeter-client\") pod \"telemeter-client-fc5d4f5c9-bksvl\" (UID: \"e9849d8f-4428-4c98-9aeb-f02cc00775d3\") " pod="openshift-monitoring/telemeter-client-fc5d4f5c9-bksvl" Sep 11 20:03:01.228695 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.228676 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e9849d8f-4428-4c98-9aeb-f02cc00775d3-federate-client-tls\") pod \"telemeter-client-fc5d4f5c9-bksvl\" (UID: \"e9849d8f-4428-4c98-9aeb-f02cc00775d3\") " pod="openshift-monitoring/telemeter-client-fc5d4f5c9-bksvl" Sep 11 20:03:01.228746 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.228700 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e9849d8f-4428-4c98-9aeb-f02cc00775d3-metrics-client-ca\") pod \"telemeter-client-fc5d4f5c9-bksvl\" (UID: \"e9849d8f-4428-4c98-9aeb-f02cc00775d3\") " pod="openshift-monitoring/telemeter-client-fc5d4f5c9-bksvl" Sep 11 20:03:01.329634 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.329605 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e9849d8f-4428-4c98-9aeb-f02cc00775d3-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-fc5d4f5c9-bksvl\" (UID: \"e9849d8f-4428-4c98-9aeb-f02cc00775d3\") " pod="openshift-monitoring/telemeter-client-fc5d4f5c9-bksvl" Sep 11 20:03:01.329775 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.329641 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e9849d8f-4428-4c98-9aeb-f02cc00775d3-secret-telemeter-client\") pod \"telemeter-client-fc5d4f5c9-bksvl\" (UID: \"e9849d8f-4428-4c98-9aeb-f02cc00775d3\") " pod="openshift-monitoring/telemeter-client-fc5d4f5c9-bksvl" Sep 11 20:03:01.329775 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.329667 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e9849d8f-4428-4c98-9aeb-f02cc00775d3-federate-client-tls\") pod \"telemeter-client-fc5d4f5c9-bksvl\" (UID: \"e9849d8f-4428-4c98-9aeb-f02cc00775d3\") " pod="openshift-monitoring/telemeter-client-fc5d4f5c9-bksvl" Sep 11 20:03:01.329866 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.329791 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e9849d8f-4428-4c98-9aeb-f02cc00775d3-metrics-client-ca\") pod \"telemeter-client-fc5d4f5c9-bksvl\" (UID: \"e9849d8f-4428-4c98-9aeb-f02cc00775d3\") " pod="openshift-monitoring/telemeter-client-fc5d4f5c9-bksvl" Sep 11 20:03:01.329866 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.329860 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9849d8f-4428-4c98-9aeb-f02cc00775d3-serving-certs-ca-bundle\") pod \"telemeter-client-fc5d4f5c9-bksvl\" (UID: \"e9849d8f-4428-4c98-9aeb-f02cc00775d3\") " pod="openshift-monitoring/telemeter-client-fc5d4f5c9-bksvl" Sep 11 20:03:01.329958 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.329893 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn24p\" (UniqueName: \"kubernetes.io/projected/e9849d8f-4428-4c98-9aeb-f02cc00775d3-kube-api-access-rn24p\") pod \"telemeter-client-fc5d4f5c9-bksvl\" (UID: \"e9849d8f-4428-4c98-9aeb-f02cc00775d3\") " pod="openshift-monitoring/telemeter-client-fc5d4f5c9-bksvl" Sep 11 20:03:01.329958 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.329922 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e9849d8f-4428-4c98-9aeb-f02cc00775d3-telemeter-client-tls\") pod \"telemeter-client-fc5d4f5c9-bksvl\" (UID: \"e9849d8f-4428-4c98-9aeb-f02cc00775d3\") " pod="openshift-monitoring/telemeter-client-fc5d4f5c9-bksvl" Sep 11 20:03:01.330044 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.329961 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9849d8f-4428-4c98-9aeb-f02cc00775d3-telemeter-trusted-ca-bundle\") pod \"telemeter-client-fc5d4f5c9-bksvl\" (UID: \"e9849d8f-4428-4c98-9aeb-f02cc00775d3\") " pod="openshift-monitoring/telemeter-client-fc5d4f5c9-bksvl" Sep 11 20:03:01.330650 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.330621 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9849d8f-4428-4c98-9aeb-f02cc00775d3-serving-certs-ca-bundle\") pod \"telemeter-client-fc5d4f5c9-bksvl\" (UID: \"e9849d8f-4428-4c98-9aeb-f02cc00775d3\") " pod="openshift-monitoring/telemeter-client-fc5d4f5c9-bksvl" Sep 11 20:03:01.330650 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.330654 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e9849d8f-4428-4c98-9aeb-f02cc00775d3-metrics-client-ca\") pod \"telemeter-client-fc5d4f5c9-bksvl\" (UID: \"e9849d8f-4428-4c98-9aeb-f02cc00775d3\") " pod="openshift-monitoring/telemeter-client-fc5d4f5c9-bksvl" Sep 11 20:03:01.330856 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.330836 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9849d8f-4428-4c98-9aeb-f02cc00775d3-telemeter-trusted-ca-bundle\") pod \"telemeter-client-fc5d4f5c9-bksvl\" (UID: \"e9849d8f-4428-4c98-9aeb-f02cc00775d3\") " pod="openshift-monitoring/telemeter-client-fc5d4f5c9-bksvl" Sep 11 20:03:01.332243 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.332219 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e9849d8f-4428-4c98-9aeb-f02cc00775d3-telemeter-client-tls\") pod \"telemeter-client-fc5d4f5c9-bksvl\" (UID: \"e9849d8f-4428-4c98-9aeb-f02cc00775d3\") " pod="openshift-monitoring/telemeter-client-fc5d4f5c9-bksvl" Sep 11 20:03:01.332347 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.332321 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e9849d8f-4428-4c98-9aeb-f02cc00775d3-secret-telemeter-client\") pod \"telemeter-client-fc5d4f5c9-bksvl\" (UID: \"e9849d8f-4428-4c98-9aeb-f02cc00775d3\") " pod="openshift-monitoring/telemeter-client-fc5d4f5c9-bksvl" Sep 11 20:03:01.332467 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.332446 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e9849d8f-4428-4c98-9aeb-f02cc00775d3-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-fc5d4f5c9-bksvl\" (UID: \"e9849d8f-4428-4c98-9aeb-f02cc00775d3\") " pod="openshift-monitoring/telemeter-client-fc5d4f5c9-bksvl" Sep 11 20:03:01.332508 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.332461 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e9849d8f-4428-4c98-9aeb-f02cc00775d3-federate-client-tls\") pod \"telemeter-client-fc5d4f5c9-bksvl\" (UID: \"e9849d8f-4428-4c98-9aeb-f02cc00775d3\") " pod="openshift-monitoring/telemeter-client-fc5d4f5c9-bksvl" Sep 11 20:03:01.337325 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.337305 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn24p\" (UniqueName: \"kubernetes.io/projected/e9849d8f-4428-4c98-9aeb-f02cc00775d3-kube-api-access-rn24p\") pod \"telemeter-client-fc5d4f5c9-bksvl\" (UID: \"e9849d8f-4428-4c98-9aeb-f02cc00775d3\") " pod="openshift-monitoring/telemeter-client-fc5d4f5c9-bksvl" Sep 11 20:03:01.400946 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.400868 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 20:03:01.401204 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.401154 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="71e7ab2f-2146-48a4-a9a0-b6ac3de11385" containerName="prometheus" containerID="cri-o://4487c4d66faf952bccf5a2bd4d15a60ae982b3a15bcf7bfa7a0187d307cc19d2" gracePeriod=600 Sep 11 20:03:01.401285 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.401205 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="71e7ab2f-2146-48a4-a9a0-b6ac3de11385" containerName="kube-rbac-proxy-thanos" containerID="cri-o://1e26caaa2e13b8294f8c63134fac5d98860699d9d18175113d22919b02b72e81" gracePeriod=600 Sep 11 20:03:01.401285 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.401208 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="71e7ab2f-2146-48a4-a9a0-b6ac3de11385" containerName="kube-rbac-proxy-web" containerID="cri-o://c6f7823a9b1c8f43ddc1bccbbe4e12f3bc445ae16a4f9836609282adb6b8b055" gracePeriod=600 Sep 11 20:03:01.401285 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.401161 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="71e7ab2f-2146-48a4-a9a0-b6ac3de11385" containerName="kube-rbac-proxy" containerID="cri-o://8be7262fa9233d57eaf437bb764aad9813b8595a143f36d393faaff8651e22a2" gracePeriod=600 Sep 11 20:03:01.401439 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.401162 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="71e7ab2f-2146-48a4-a9a0-b6ac3de11385" containerName="thanos-sidecar" containerID="cri-o://67da36248892982f4627891345c94bbbdd9f80fdbffd1df19b33417829189d75" gracePeriod=600 Sep 11 20:03:01.401514 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.401189 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="71e7ab2f-2146-48a4-a9a0-b6ac3de11385" containerName="config-reloader" containerID="cri-o://850eec169eb8fb81640c4dbcef362bdd0da6665e52975d0df320f24d7584571b" gracePeriod=600 Sep 11 20:03:01.403317 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.403113 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-fc5d4f5c9-bksvl" Sep 11 20:03:01.646851 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.646825 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:03:01.703925 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.703836 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-fc5d4f5c9-bksvl"] Sep 11 20:03:01.706754 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:03:01.706720 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9849d8f_4428_4c98_9aeb_f02cc00775d3.slice/crio-2e59d53904deeefa4dd02a473a117f330d4191646ff1238fa95e446e4e162aa0 WatchSource:0}: Error finding container 2e59d53904deeefa4dd02a473a117f330d4191646ff1238fa95e446e4e162aa0: Status 404 returned error can't find the container with id 2e59d53904deeefa4dd02a473a117f330d4191646ff1238fa95e446e4e162aa0 Sep 11 20:03:01.733953 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.733925 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-web-config\") pod \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " Sep 11 20:03:01.734077 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.733965 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-tls-assets\") pod \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " Sep 11 20:03:01.734077 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.733989 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-secret-prometheus-k8s-tls\") pod \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " Sep 11 20:03:01.734077 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.734021 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-configmap-metrics-client-ca\") pod \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " Sep 11 20:03:01.734077 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.734049 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-secret-grpc-tls\") pod \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " Sep 11 20:03:01.734403 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.734094 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-prometheus-k8s-rulefiles-2\") pod \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " Sep 11 20:03:01.734403 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.734131 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-config\") pod \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " Sep 11 20:03:01.734403 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.734166 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-prometheus-k8s-rulefiles-0\") pod \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " Sep 11 20:03:01.734403 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.734210 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-prometheus-k8s-db\") pod \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " Sep 11 20:03:01.734403 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.734236 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-secret-kube-rbac-proxy\") pod \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " Sep 11 20:03:01.734403 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.734268 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-thanos-prometheus-http-client-file\") pod \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " Sep 11 20:03:01.734403 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.734319 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-config-out\") pod \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " Sep 11 20:03:01.734403 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.734350 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-configmap-kubelet-serving-ca-bundle\") pod \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " Sep 11 20:03:01.734403 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.734381 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-prometheus-k8s-rulefiles-1\") pod \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " Sep 11 20:03:01.734884 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.734412 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-configmap-serving-certs-ca-bundle\") pod \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " Sep 11 20:03:01.734884 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.734420 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "71e7ab2f-2146-48a4-a9a0-b6ac3de11385" (UID: "71e7ab2f-2146-48a4-a9a0-b6ac3de11385"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:03:01.734884 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.734448 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ltlbl\" (UniqueName: \"kubernetes.io/projected/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-kube-api-access-ltlbl\") pod \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " Sep 11 20:03:01.734884 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.734473 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-secret-metrics-client-certs\") pod \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " Sep 11 20:03:01.734884 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.734506 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-prometheus-trusted-ca-bundle\") pod \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " Sep 11 20:03:01.734884 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.734532 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " Sep 11 20:03:01.734884 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.734559 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\" (UID: \"71e7ab2f-2146-48a4-a9a0-b6ac3de11385\") " Sep 11 20:03:01.734884 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.734837 2546 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-configmap-metrics-client-ca\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:03:01.735917 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.735487 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "71e7ab2f-2146-48a4-a9a0-b6ac3de11385" (UID: "71e7ab2f-2146-48a4-a9a0-b6ac3de11385"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:03:01.736540 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.736479 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "71e7ab2f-2146-48a4-a9a0-b6ac3de11385" (UID: "71e7ab2f-2146-48a4-a9a0-b6ac3de11385"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:03:01.736540 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.736516 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "71e7ab2f-2146-48a4-a9a0-b6ac3de11385" (UID: "71e7ab2f-2146-48a4-a9a0-b6ac3de11385"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 20:03:01.737080 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.737023 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "71e7ab2f-2146-48a4-a9a0-b6ac3de11385" (UID: "71e7ab2f-2146-48a4-a9a0-b6ac3de11385"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:03:01.737253 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.737224 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "71e7ab2f-2146-48a4-a9a0-b6ac3de11385" (UID: "71e7ab2f-2146-48a4-a9a0-b6ac3de11385"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:03:01.737440 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.737409 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "71e7ab2f-2146-48a4-a9a0-b6ac3de11385" (UID: "71e7ab2f-2146-48a4-a9a0-b6ac3de11385"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:03:01.737440 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.737425 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "71e7ab2f-2146-48a4-a9a0-b6ac3de11385" (UID: "71e7ab2f-2146-48a4-a9a0-b6ac3de11385"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:03:01.737569 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.737460 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "71e7ab2f-2146-48a4-a9a0-b6ac3de11385" (UID: "71e7ab2f-2146-48a4-a9a0-b6ac3de11385"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:03:01.737569 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.737550 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-config" (OuterVolumeSpecName: "config") pod "71e7ab2f-2146-48a4-a9a0-b6ac3de11385" (UID: "71e7ab2f-2146-48a4-a9a0-b6ac3de11385"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 20:03:01.737870 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.737842 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "71e7ab2f-2146-48a4-a9a0-b6ac3de11385" (UID: "71e7ab2f-2146-48a4-a9a0-b6ac3de11385"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 20:03:01.738109 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.738082 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "71e7ab2f-2146-48a4-a9a0-b6ac3de11385" (UID: "71e7ab2f-2146-48a4-a9a0-b6ac3de11385"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:03:01.739011 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.738976 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "71e7ab2f-2146-48a4-a9a0-b6ac3de11385" (UID: "71e7ab2f-2146-48a4-a9a0-b6ac3de11385"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 20:03:01.739520 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.739491 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "71e7ab2f-2146-48a4-a9a0-b6ac3de11385" (UID: "71e7ab2f-2146-48a4-a9a0-b6ac3de11385"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 20:03:01.739832 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.739797 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "71e7ab2f-2146-48a4-a9a0-b6ac3de11385" (UID: "71e7ab2f-2146-48a4-a9a0-b6ac3de11385"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 20:03:01.739949 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.739841 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "71e7ab2f-2146-48a4-a9a0-b6ac3de11385" (UID: "71e7ab2f-2146-48a4-a9a0-b6ac3de11385"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 20:03:01.740445 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.740422 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "71e7ab2f-2146-48a4-a9a0-b6ac3de11385" (UID: "71e7ab2f-2146-48a4-a9a0-b6ac3de11385"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 20:03:01.740628 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.740612 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-kube-api-access-ltlbl" (OuterVolumeSpecName: "kube-api-access-ltlbl") pod "71e7ab2f-2146-48a4-a9a0-b6ac3de11385" (UID: "71e7ab2f-2146-48a4-a9a0-b6ac3de11385"). InnerVolumeSpecName "kube-api-access-ltlbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:03:01.740883 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.740870 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-config-out" (OuterVolumeSpecName: "config-out") pod "71e7ab2f-2146-48a4-a9a0-b6ac3de11385" (UID: "71e7ab2f-2146-48a4-a9a0-b6ac3de11385"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:03:01.747589 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.747551 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-web-config" (OuterVolumeSpecName: "web-config") pod "71e7ab2f-2146-48a4-a9a0-b6ac3de11385" (UID: "71e7ab2f-2146-48a4-a9a0-b6ac3de11385"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 20:03:01.835825 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.835789 2546 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:03:01.835825 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.835820 2546 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-config\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:03:01.835825 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.835832 2546 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:03:01.836049 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.835841 2546 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-prometheus-k8s-db\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:03:01.836049 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.835849 2546 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-secret-kube-rbac-proxy\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:03:01.836049 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.835858 2546 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-thanos-prometheus-http-client-file\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:03:01.836049 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.835867 2546 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-config-out\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:03:01.836049 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.835877 2546 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:03:01.836049 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.835885 2546 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:03:01.836049 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.835893 2546 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:03:01.836049 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.835902 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltlbl\" (UniqueName: \"kubernetes.io/projected/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-kube-api-access-ltlbl\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:03:01.836049 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.835912 2546 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-secret-metrics-client-certs\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:03:01.836049 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.835920 2546 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-prometheus-trusted-ca-bundle\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:03:01.836049 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.835928 2546 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:03:01.836049 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.835937 2546 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:03:01.836049 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.835945 2546 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-web-config\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:03:01.836049 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.835954 2546 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-tls-assets\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:03:01.836049 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.835961 2546 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-secret-prometheus-k8s-tls\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:03:01.836049 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:01.835970 2546 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/71e7ab2f-2146-48a4-a9a0-b6ac3de11385-secret-grpc-tls\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:03:02.180785 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.180759 2546 generic.go:356] "Generic (PLEG): container finished" podID="71e7ab2f-2146-48a4-a9a0-b6ac3de11385" containerID="1e26caaa2e13b8294f8c63134fac5d98860699d9d18175113d22919b02b72e81" exitCode=0 Sep 11 20:03:02.180785 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.180782 2546 generic.go:356] "Generic (PLEG): container finished" podID="71e7ab2f-2146-48a4-a9a0-b6ac3de11385" containerID="8be7262fa9233d57eaf437bb764aad9813b8595a143f36d393faaff8651e22a2" exitCode=0 Sep 11 20:03:02.180785 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.180788 2546 generic.go:356] "Generic (PLEG): container finished" podID="71e7ab2f-2146-48a4-a9a0-b6ac3de11385" containerID="c6f7823a9b1c8f43ddc1bccbbe4e12f3bc445ae16a4f9836609282adb6b8b055" exitCode=0 Sep 11 20:03:02.181278 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.180794 2546 generic.go:356] "Generic (PLEG): container finished" podID="71e7ab2f-2146-48a4-a9a0-b6ac3de11385" containerID="67da36248892982f4627891345c94bbbdd9f80fdbffd1df19b33417829189d75" exitCode=0 Sep 11 20:03:02.181278 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.180799 2546 generic.go:356] "Generic (PLEG): container finished" podID="71e7ab2f-2146-48a4-a9a0-b6ac3de11385" containerID="850eec169eb8fb81640c4dbcef362bdd0da6665e52975d0df320f24d7584571b" exitCode=0 Sep 11 20:03:02.181278 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.180804 2546 generic.go:356] "Generic (PLEG): container finished" podID="71e7ab2f-2146-48a4-a9a0-b6ac3de11385" containerID="4487c4d66faf952bccf5a2bd4d15a60ae982b3a15bcf7bfa7a0187d307cc19d2" exitCode=0 Sep 11 20:03:02.181278 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.180831 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"71e7ab2f-2146-48a4-a9a0-b6ac3de11385","Type":"ContainerDied","Data":"1e26caaa2e13b8294f8c63134fac5d98860699d9d18175113d22919b02b72e81"} Sep 11 20:03:02.181278 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.180867 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:03:02.181278 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.180918 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"71e7ab2f-2146-48a4-a9a0-b6ac3de11385","Type":"ContainerDied","Data":"8be7262fa9233d57eaf437bb764aad9813b8595a143f36d393faaff8651e22a2"} Sep 11 20:03:02.181278 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.180961 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"71e7ab2f-2146-48a4-a9a0-b6ac3de11385","Type":"ContainerDied","Data":"c6f7823a9b1c8f43ddc1bccbbe4e12f3bc445ae16a4f9836609282adb6b8b055"} Sep 11 20:03:02.181278 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.180989 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"71e7ab2f-2146-48a4-a9a0-b6ac3de11385","Type":"ContainerDied","Data":"67da36248892982f4627891345c94bbbdd9f80fdbffd1df19b33417829189d75"} Sep 11 20:03:02.181278 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.181013 2546 scope.go:117] "RemoveContainer" containerID="1e26caaa2e13b8294f8c63134fac5d98860699d9d18175113d22919b02b72e81" Sep 11 20:03:02.181278 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.181028 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"71e7ab2f-2146-48a4-a9a0-b6ac3de11385","Type":"ContainerDied","Data":"850eec169eb8fb81640c4dbcef362bdd0da6665e52975d0df320f24d7584571b"} Sep 11 20:03:02.181278 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.181067 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"71e7ab2f-2146-48a4-a9a0-b6ac3de11385","Type":"ContainerDied","Data":"4487c4d66faf952bccf5a2bd4d15a60ae982b3a15bcf7bfa7a0187d307cc19d2"} Sep 11 20:03:02.181278 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.181105 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"71e7ab2f-2146-48a4-a9a0-b6ac3de11385","Type":"ContainerDied","Data":"54ed21b2934e04938de9cf5f5ae400df1581e3f32d82cfa00c2b6892174bf990"} Sep 11 20:03:02.182191 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.182170 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-fc5d4f5c9-bksvl" event={"ID":"e9849d8f-4428-4c98-9aeb-f02cc00775d3","Type":"ContainerStarted","Data":"2e59d53904deeefa4dd02a473a117f330d4191646ff1238fa95e446e4e162aa0"} Sep 11 20:03:02.189416 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.189399 2546 scope.go:117] "RemoveContainer" containerID="8be7262fa9233d57eaf437bb764aad9813b8595a143f36d393faaff8651e22a2" Sep 11 20:03:02.196209 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.196191 2546 scope.go:117] "RemoveContainer" containerID="c6f7823a9b1c8f43ddc1bccbbe4e12f3bc445ae16a4f9836609282adb6b8b055" Sep 11 20:03:02.202725 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.202709 2546 scope.go:117] "RemoveContainer" containerID="67da36248892982f4627891345c94bbbdd9f80fdbffd1df19b33417829189d75" Sep 11 20:03:02.205541 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.205522 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 20:03:02.209314 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.209291 2546 scope.go:117] "RemoveContainer" containerID="850eec169eb8fb81640c4dbcef362bdd0da6665e52975d0df320f24d7584571b" Sep 11 20:03:02.210796 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.210776 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 20:03:02.215654 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.215638 2546 scope.go:117] "RemoveContainer" containerID="4487c4d66faf952bccf5a2bd4d15a60ae982b3a15bcf7bfa7a0187d307cc19d2" Sep 11 20:03:02.222360 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.222345 2546 scope.go:117] "RemoveContainer" containerID="4d7aa46e3dc629a5238ca22dc98949c1627827ca573e5c5fba3b700cd5d6098b" Sep 11 20:03:02.228788 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.228771 2546 scope.go:117] "RemoveContainer" containerID="1e26caaa2e13b8294f8c63134fac5d98860699d9d18175113d22919b02b72e81" Sep 11 20:03:02.229012 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:03:02.228996 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e26caaa2e13b8294f8c63134fac5d98860699d9d18175113d22919b02b72e81\": container with ID starting with 1e26caaa2e13b8294f8c63134fac5d98860699d9d18175113d22919b02b72e81 not found: ID does not exist" containerID="1e26caaa2e13b8294f8c63134fac5d98860699d9d18175113d22919b02b72e81" Sep 11 20:03:02.229058 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.229018 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e26caaa2e13b8294f8c63134fac5d98860699d9d18175113d22919b02b72e81"} err="failed to get container status \"1e26caaa2e13b8294f8c63134fac5d98860699d9d18175113d22919b02b72e81\": rpc error: code = NotFound desc = could not find container \"1e26caaa2e13b8294f8c63134fac5d98860699d9d18175113d22919b02b72e81\": container with ID starting with 1e26caaa2e13b8294f8c63134fac5d98860699d9d18175113d22919b02b72e81 not found: ID does not exist" Sep 11 20:03:02.229058 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.229032 2546 scope.go:117] "RemoveContainer" containerID="8be7262fa9233d57eaf437bb764aad9813b8595a143f36d393faaff8651e22a2" Sep 11 20:03:02.229280 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:03:02.229263 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8be7262fa9233d57eaf437bb764aad9813b8595a143f36d393faaff8651e22a2\": container with ID starting with 8be7262fa9233d57eaf437bb764aad9813b8595a143f36d393faaff8651e22a2 not found: ID does not exist" containerID="8be7262fa9233d57eaf437bb764aad9813b8595a143f36d393faaff8651e22a2" Sep 11 20:03:02.229389 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.229289 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8be7262fa9233d57eaf437bb764aad9813b8595a143f36d393faaff8651e22a2"} err="failed to get container status \"8be7262fa9233d57eaf437bb764aad9813b8595a143f36d393faaff8651e22a2\": rpc error: code = NotFound desc = could not find container \"8be7262fa9233d57eaf437bb764aad9813b8595a143f36d393faaff8651e22a2\": container with ID starting with 8be7262fa9233d57eaf437bb764aad9813b8595a143f36d393faaff8651e22a2 not found: ID does not exist" Sep 11 20:03:02.229389 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.229313 2546 scope.go:117] "RemoveContainer" containerID="c6f7823a9b1c8f43ddc1bccbbe4e12f3bc445ae16a4f9836609282adb6b8b055" Sep 11 20:03:02.229545 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:03:02.229530 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6f7823a9b1c8f43ddc1bccbbe4e12f3bc445ae16a4f9836609282adb6b8b055\": container with ID starting with c6f7823a9b1c8f43ddc1bccbbe4e12f3bc445ae16a4f9836609282adb6b8b055 not found: ID does not exist" containerID="c6f7823a9b1c8f43ddc1bccbbe4e12f3bc445ae16a4f9836609282adb6b8b055" Sep 11 20:03:02.229603 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.229549 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6f7823a9b1c8f43ddc1bccbbe4e12f3bc445ae16a4f9836609282adb6b8b055"} err="failed to get container status \"c6f7823a9b1c8f43ddc1bccbbe4e12f3bc445ae16a4f9836609282adb6b8b055\": rpc error: code = NotFound desc = could not find container \"c6f7823a9b1c8f43ddc1bccbbe4e12f3bc445ae16a4f9836609282adb6b8b055\": container with ID starting with c6f7823a9b1c8f43ddc1bccbbe4e12f3bc445ae16a4f9836609282adb6b8b055 not found: ID does not exist" Sep 11 20:03:02.229603 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.229563 2546 scope.go:117] "RemoveContainer" containerID="67da36248892982f4627891345c94bbbdd9f80fdbffd1df19b33417829189d75" Sep 11 20:03:02.229786 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:03:02.229769 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67da36248892982f4627891345c94bbbdd9f80fdbffd1df19b33417829189d75\": container with ID starting with 67da36248892982f4627891345c94bbbdd9f80fdbffd1df19b33417829189d75 not found: ID does not exist" containerID="67da36248892982f4627891345c94bbbdd9f80fdbffd1df19b33417829189d75" Sep 11 20:03:02.229850 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.229793 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67da36248892982f4627891345c94bbbdd9f80fdbffd1df19b33417829189d75"} err="failed to get container status \"67da36248892982f4627891345c94bbbdd9f80fdbffd1df19b33417829189d75\": rpc error: code = NotFound desc = could not find container \"67da36248892982f4627891345c94bbbdd9f80fdbffd1df19b33417829189d75\": container with ID starting with 67da36248892982f4627891345c94bbbdd9f80fdbffd1df19b33417829189d75 not found: ID does not exist" Sep 11 20:03:02.229850 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.229812 2546 scope.go:117] "RemoveContainer" containerID="850eec169eb8fb81640c4dbcef362bdd0da6665e52975d0df320f24d7584571b" Sep 11 20:03:02.230061 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:03:02.230045 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"850eec169eb8fb81640c4dbcef362bdd0da6665e52975d0df320f24d7584571b\": container with ID starting with 850eec169eb8fb81640c4dbcef362bdd0da6665e52975d0df320f24d7584571b not found: ID does not exist" containerID="850eec169eb8fb81640c4dbcef362bdd0da6665e52975d0df320f24d7584571b" Sep 11 20:03:02.230122 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.230066 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"850eec169eb8fb81640c4dbcef362bdd0da6665e52975d0df320f24d7584571b"} err="failed to get container status \"850eec169eb8fb81640c4dbcef362bdd0da6665e52975d0df320f24d7584571b\": rpc error: code = NotFound desc = could not find container \"850eec169eb8fb81640c4dbcef362bdd0da6665e52975d0df320f24d7584571b\": container with ID starting with 850eec169eb8fb81640c4dbcef362bdd0da6665e52975d0df320f24d7584571b not found: ID does not exist" Sep 11 20:03:02.230122 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.230083 2546 scope.go:117] "RemoveContainer" containerID="4487c4d66faf952bccf5a2bd4d15a60ae982b3a15bcf7bfa7a0187d307cc19d2" Sep 11 20:03:02.230288 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:03:02.230273 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4487c4d66faf952bccf5a2bd4d15a60ae982b3a15bcf7bfa7a0187d307cc19d2\": container with ID starting with 4487c4d66faf952bccf5a2bd4d15a60ae982b3a15bcf7bfa7a0187d307cc19d2 not found: ID does not exist" containerID="4487c4d66faf952bccf5a2bd4d15a60ae982b3a15bcf7bfa7a0187d307cc19d2" Sep 11 20:03:02.230347 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.230294 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4487c4d66faf952bccf5a2bd4d15a60ae982b3a15bcf7bfa7a0187d307cc19d2"} err="failed to get container status \"4487c4d66faf952bccf5a2bd4d15a60ae982b3a15bcf7bfa7a0187d307cc19d2\": rpc error: code = NotFound desc = could not find container \"4487c4d66faf952bccf5a2bd4d15a60ae982b3a15bcf7bfa7a0187d307cc19d2\": container with ID starting with 4487c4d66faf952bccf5a2bd4d15a60ae982b3a15bcf7bfa7a0187d307cc19d2 not found: ID does not exist" Sep 11 20:03:02.230347 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.230311 2546 scope.go:117] "RemoveContainer" containerID="4d7aa46e3dc629a5238ca22dc98949c1627827ca573e5c5fba3b700cd5d6098b" Sep 11 20:03:02.230547 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:03:02.230531 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d7aa46e3dc629a5238ca22dc98949c1627827ca573e5c5fba3b700cd5d6098b\": container with ID starting with 4d7aa46e3dc629a5238ca22dc98949c1627827ca573e5c5fba3b700cd5d6098b not found: ID does not exist" containerID="4d7aa46e3dc629a5238ca22dc98949c1627827ca573e5c5fba3b700cd5d6098b" Sep 11 20:03:02.230611 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.230551 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d7aa46e3dc629a5238ca22dc98949c1627827ca573e5c5fba3b700cd5d6098b"} err="failed to get container status \"4d7aa46e3dc629a5238ca22dc98949c1627827ca573e5c5fba3b700cd5d6098b\": rpc error: code = NotFound desc = could not find container \"4d7aa46e3dc629a5238ca22dc98949c1627827ca573e5c5fba3b700cd5d6098b\": container with ID starting with 4d7aa46e3dc629a5238ca22dc98949c1627827ca573e5c5fba3b700cd5d6098b not found: ID does not exist" Sep 11 20:03:02.230611 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.230563 2546 scope.go:117] "RemoveContainer" containerID="1e26caaa2e13b8294f8c63134fac5d98860699d9d18175113d22919b02b72e81" Sep 11 20:03:02.230795 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.230780 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e26caaa2e13b8294f8c63134fac5d98860699d9d18175113d22919b02b72e81"} err="failed to get container status \"1e26caaa2e13b8294f8c63134fac5d98860699d9d18175113d22919b02b72e81\": rpc error: code = NotFound desc = could not find container \"1e26caaa2e13b8294f8c63134fac5d98860699d9d18175113d22919b02b72e81\": container with ID starting with 1e26caaa2e13b8294f8c63134fac5d98860699d9d18175113d22919b02b72e81 not found: ID does not exist" Sep 11 20:03:02.230849 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.230796 2546 scope.go:117] "RemoveContainer" containerID="8be7262fa9233d57eaf437bb764aad9813b8595a143f36d393faaff8651e22a2" Sep 11 20:03:02.231019 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.231003 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8be7262fa9233d57eaf437bb764aad9813b8595a143f36d393faaff8651e22a2"} err="failed to get container status \"8be7262fa9233d57eaf437bb764aad9813b8595a143f36d393faaff8651e22a2\": rpc error: code = NotFound desc = could not find container \"8be7262fa9233d57eaf437bb764aad9813b8595a143f36d393faaff8651e22a2\": container with ID starting with 8be7262fa9233d57eaf437bb764aad9813b8595a143f36d393faaff8651e22a2 not found: ID does not exist" Sep 11 20:03:02.231056 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.231020 2546 scope.go:117] "RemoveContainer" containerID="c6f7823a9b1c8f43ddc1bccbbe4e12f3bc445ae16a4f9836609282adb6b8b055" Sep 11 20:03:02.231234 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.231218 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6f7823a9b1c8f43ddc1bccbbe4e12f3bc445ae16a4f9836609282adb6b8b055"} err="failed to get container status \"c6f7823a9b1c8f43ddc1bccbbe4e12f3bc445ae16a4f9836609282adb6b8b055\": rpc error: code = NotFound desc = could not find container \"c6f7823a9b1c8f43ddc1bccbbe4e12f3bc445ae16a4f9836609282adb6b8b055\": container with ID starting with c6f7823a9b1c8f43ddc1bccbbe4e12f3bc445ae16a4f9836609282adb6b8b055 not found: ID does not exist" Sep 11 20:03:02.231285 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.231233 2546 scope.go:117] "RemoveContainer" containerID="67da36248892982f4627891345c94bbbdd9f80fdbffd1df19b33417829189d75" Sep 11 20:03:02.231421 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.231404 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67da36248892982f4627891345c94bbbdd9f80fdbffd1df19b33417829189d75"} err="failed to get container status \"67da36248892982f4627891345c94bbbdd9f80fdbffd1df19b33417829189d75\": rpc error: code = NotFound desc = could not find container \"67da36248892982f4627891345c94bbbdd9f80fdbffd1df19b33417829189d75\": container with ID starting with 67da36248892982f4627891345c94bbbdd9f80fdbffd1df19b33417829189d75 not found: ID does not exist" Sep 11 20:03:02.231474 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.231421 2546 scope.go:117] "RemoveContainer" containerID="850eec169eb8fb81640c4dbcef362bdd0da6665e52975d0df320f24d7584571b" Sep 11 20:03:02.231701 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.231681 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"850eec169eb8fb81640c4dbcef362bdd0da6665e52975d0df320f24d7584571b"} err="failed to get container status \"850eec169eb8fb81640c4dbcef362bdd0da6665e52975d0df320f24d7584571b\": rpc error: code = NotFound desc = could not find container \"850eec169eb8fb81640c4dbcef362bdd0da6665e52975d0df320f24d7584571b\": container with ID starting with 850eec169eb8fb81640c4dbcef362bdd0da6665e52975d0df320f24d7584571b not found: ID does not exist" Sep 11 20:03:02.231750 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.231702 2546 scope.go:117] "RemoveContainer" containerID="4487c4d66faf952bccf5a2bd4d15a60ae982b3a15bcf7bfa7a0187d307cc19d2" Sep 11 20:03:02.231902 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.231886 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4487c4d66faf952bccf5a2bd4d15a60ae982b3a15bcf7bfa7a0187d307cc19d2"} err="failed to get container status \"4487c4d66faf952bccf5a2bd4d15a60ae982b3a15bcf7bfa7a0187d307cc19d2\": rpc error: code = NotFound desc = could not find container \"4487c4d66faf952bccf5a2bd4d15a60ae982b3a15bcf7bfa7a0187d307cc19d2\": container with ID starting with 4487c4d66faf952bccf5a2bd4d15a60ae982b3a15bcf7bfa7a0187d307cc19d2 not found: ID does not exist" Sep 11 20:03:02.231902 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.231901 2546 scope.go:117] "RemoveContainer" containerID="4d7aa46e3dc629a5238ca22dc98949c1627827ca573e5c5fba3b700cd5d6098b" Sep 11 20:03:02.232131 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.232111 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d7aa46e3dc629a5238ca22dc98949c1627827ca573e5c5fba3b700cd5d6098b"} err="failed to get container status \"4d7aa46e3dc629a5238ca22dc98949c1627827ca573e5c5fba3b700cd5d6098b\": rpc error: code = NotFound desc = could not find container \"4d7aa46e3dc629a5238ca22dc98949c1627827ca573e5c5fba3b700cd5d6098b\": container with ID starting with 4d7aa46e3dc629a5238ca22dc98949c1627827ca573e5c5fba3b700cd5d6098b not found: ID does not exist" Sep 11 20:03:02.232188 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.232132 2546 scope.go:117] "RemoveContainer" containerID="1e26caaa2e13b8294f8c63134fac5d98860699d9d18175113d22919b02b72e81" Sep 11 20:03:02.232396 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.232377 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e26caaa2e13b8294f8c63134fac5d98860699d9d18175113d22919b02b72e81"} err="failed to get container status \"1e26caaa2e13b8294f8c63134fac5d98860699d9d18175113d22919b02b72e81\": rpc error: code = NotFound desc = could not find container \"1e26caaa2e13b8294f8c63134fac5d98860699d9d18175113d22919b02b72e81\": container with ID starting with 1e26caaa2e13b8294f8c63134fac5d98860699d9d18175113d22919b02b72e81 not found: ID does not exist" Sep 11 20:03:02.232455 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.232404 2546 scope.go:117] "RemoveContainer" containerID="8be7262fa9233d57eaf437bb764aad9813b8595a143f36d393faaff8651e22a2" Sep 11 20:03:02.232637 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.232621 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8be7262fa9233d57eaf437bb764aad9813b8595a143f36d393faaff8651e22a2"} err="failed to get container status \"8be7262fa9233d57eaf437bb764aad9813b8595a143f36d393faaff8651e22a2\": rpc error: code = NotFound desc = could not find container \"8be7262fa9233d57eaf437bb764aad9813b8595a143f36d393faaff8651e22a2\": container with ID starting with 8be7262fa9233d57eaf437bb764aad9813b8595a143f36d393faaff8651e22a2 not found: ID does not exist" Sep 11 20:03:02.232692 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.232636 2546 scope.go:117] "RemoveContainer" containerID="c6f7823a9b1c8f43ddc1bccbbe4e12f3bc445ae16a4f9836609282adb6b8b055" Sep 11 20:03:02.232858 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.232841 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6f7823a9b1c8f43ddc1bccbbe4e12f3bc445ae16a4f9836609282adb6b8b055"} err="failed to get container status \"c6f7823a9b1c8f43ddc1bccbbe4e12f3bc445ae16a4f9836609282adb6b8b055\": rpc error: code = NotFound desc = could not find container \"c6f7823a9b1c8f43ddc1bccbbe4e12f3bc445ae16a4f9836609282adb6b8b055\": container with ID starting with c6f7823a9b1c8f43ddc1bccbbe4e12f3bc445ae16a4f9836609282adb6b8b055 not found: ID does not exist" Sep 11 20:03:02.232918 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.232858 2546 scope.go:117] "RemoveContainer" containerID="67da36248892982f4627891345c94bbbdd9f80fdbffd1df19b33417829189d75" Sep 11 20:03:02.233090 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.233073 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67da36248892982f4627891345c94bbbdd9f80fdbffd1df19b33417829189d75"} err="failed to get container status \"67da36248892982f4627891345c94bbbdd9f80fdbffd1df19b33417829189d75\": rpc error: code = NotFound desc = could not find container \"67da36248892982f4627891345c94bbbdd9f80fdbffd1df19b33417829189d75\": container with ID starting with 67da36248892982f4627891345c94bbbdd9f80fdbffd1df19b33417829189d75 not found: ID does not exist" Sep 11 20:03:02.233135 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.233091 2546 scope.go:117] "RemoveContainer" containerID="850eec169eb8fb81640c4dbcef362bdd0da6665e52975d0df320f24d7584571b" Sep 11 20:03:02.233320 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.233297 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"850eec169eb8fb81640c4dbcef362bdd0da6665e52975d0df320f24d7584571b"} err="failed to get container status \"850eec169eb8fb81640c4dbcef362bdd0da6665e52975d0df320f24d7584571b\": rpc error: code = NotFound desc = could not find container \"850eec169eb8fb81640c4dbcef362bdd0da6665e52975d0df320f24d7584571b\": container with ID starting with 850eec169eb8fb81640c4dbcef362bdd0da6665e52975d0df320f24d7584571b not found: ID does not exist" Sep 11 20:03:02.233382 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.233319 2546 scope.go:117] "RemoveContainer" containerID="4487c4d66faf952bccf5a2bd4d15a60ae982b3a15bcf7bfa7a0187d307cc19d2" Sep 11 20:03:02.233597 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.233559 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4487c4d66faf952bccf5a2bd4d15a60ae982b3a15bcf7bfa7a0187d307cc19d2"} err="failed to get container status \"4487c4d66faf952bccf5a2bd4d15a60ae982b3a15bcf7bfa7a0187d307cc19d2\": rpc error: code = NotFound desc = could not find container \"4487c4d66faf952bccf5a2bd4d15a60ae982b3a15bcf7bfa7a0187d307cc19d2\": container with ID starting with 4487c4d66faf952bccf5a2bd4d15a60ae982b3a15bcf7bfa7a0187d307cc19d2 not found: ID does not exist" Sep 11 20:03:02.233713 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.233598 2546 scope.go:117] "RemoveContainer" containerID="4d7aa46e3dc629a5238ca22dc98949c1627827ca573e5c5fba3b700cd5d6098b" Sep 11 20:03:02.233908 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.233881 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d7aa46e3dc629a5238ca22dc98949c1627827ca573e5c5fba3b700cd5d6098b"} err="failed to get container status \"4d7aa46e3dc629a5238ca22dc98949c1627827ca573e5c5fba3b700cd5d6098b\": rpc error: code = NotFound desc = could not find container \"4d7aa46e3dc629a5238ca22dc98949c1627827ca573e5c5fba3b700cd5d6098b\": container with ID starting with 4d7aa46e3dc629a5238ca22dc98949c1627827ca573e5c5fba3b700cd5d6098b not found: ID does not exist" Sep 11 20:03:02.233908 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.233908 2546 scope.go:117] "RemoveContainer" containerID="1e26caaa2e13b8294f8c63134fac5d98860699d9d18175113d22919b02b72e81" Sep 11 20:03:02.234326 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.234303 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e26caaa2e13b8294f8c63134fac5d98860699d9d18175113d22919b02b72e81"} err="failed to get container status \"1e26caaa2e13b8294f8c63134fac5d98860699d9d18175113d22919b02b72e81\": rpc error: code = NotFound desc = could not find container \"1e26caaa2e13b8294f8c63134fac5d98860699d9d18175113d22919b02b72e81\": container with ID starting with 1e26caaa2e13b8294f8c63134fac5d98860699d9d18175113d22919b02b72e81 not found: ID does not exist" Sep 11 20:03:02.234462 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.234328 2546 scope.go:117] "RemoveContainer" containerID="8be7262fa9233d57eaf437bb764aad9813b8595a143f36d393faaff8651e22a2" Sep 11 20:03:02.234711 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.234690 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8be7262fa9233d57eaf437bb764aad9813b8595a143f36d393faaff8651e22a2"} err="failed to get container status \"8be7262fa9233d57eaf437bb764aad9813b8595a143f36d393faaff8651e22a2\": rpc error: code = NotFound desc = could not find container \"8be7262fa9233d57eaf437bb764aad9813b8595a143f36d393faaff8651e22a2\": container with ID starting with 8be7262fa9233d57eaf437bb764aad9813b8595a143f36d393faaff8651e22a2 not found: ID does not exist" Sep 11 20:03:02.234711 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.234710 2546 scope.go:117] "RemoveContainer" containerID="c6f7823a9b1c8f43ddc1bccbbe4e12f3bc445ae16a4f9836609282adb6b8b055" Sep 11 20:03:02.234975 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.234947 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6f7823a9b1c8f43ddc1bccbbe4e12f3bc445ae16a4f9836609282adb6b8b055"} err="failed to get container status \"c6f7823a9b1c8f43ddc1bccbbe4e12f3bc445ae16a4f9836609282adb6b8b055\": rpc error: code = NotFound desc = could not find container \"c6f7823a9b1c8f43ddc1bccbbe4e12f3bc445ae16a4f9836609282adb6b8b055\": container with ID starting with c6f7823a9b1c8f43ddc1bccbbe4e12f3bc445ae16a4f9836609282adb6b8b055 not found: ID does not exist" Sep 11 20:03:02.235041 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.234977 2546 scope.go:117] "RemoveContainer" containerID="67da36248892982f4627891345c94bbbdd9f80fdbffd1df19b33417829189d75" Sep 11 20:03:02.235236 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.235212 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67da36248892982f4627891345c94bbbdd9f80fdbffd1df19b33417829189d75"} err="failed to get container status \"67da36248892982f4627891345c94bbbdd9f80fdbffd1df19b33417829189d75\": rpc error: code = NotFound desc = could not find container \"67da36248892982f4627891345c94bbbdd9f80fdbffd1df19b33417829189d75\": container with ID starting with 67da36248892982f4627891345c94bbbdd9f80fdbffd1df19b33417829189d75 not found: ID does not exist" Sep 11 20:03:02.235287 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.235239 2546 scope.go:117] "RemoveContainer" containerID="850eec169eb8fb81640c4dbcef362bdd0da6665e52975d0df320f24d7584571b" Sep 11 20:03:02.235457 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.235439 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"850eec169eb8fb81640c4dbcef362bdd0da6665e52975d0df320f24d7584571b"} err="failed to get container status \"850eec169eb8fb81640c4dbcef362bdd0da6665e52975d0df320f24d7584571b\": rpc error: code = NotFound desc = could not find container \"850eec169eb8fb81640c4dbcef362bdd0da6665e52975d0df320f24d7584571b\": container with ID starting with 850eec169eb8fb81640c4dbcef362bdd0da6665e52975d0df320f24d7584571b not found: ID does not exist" Sep 11 20:03:02.235538 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.235458 2546 scope.go:117] "RemoveContainer" containerID="4487c4d66faf952bccf5a2bd4d15a60ae982b3a15bcf7bfa7a0187d307cc19d2" Sep 11 20:03:02.235748 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.235724 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4487c4d66faf952bccf5a2bd4d15a60ae982b3a15bcf7bfa7a0187d307cc19d2"} err="failed to get container status \"4487c4d66faf952bccf5a2bd4d15a60ae982b3a15bcf7bfa7a0187d307cc19d2\": rpc error: code = NotFound desc = could not find container \"4487c4d66faf952bccf5a2bd4d15a60ae982b3a15bcf7bfa7a0187d307cc19d2\": container with ID starting with 4487c4d66faf952bccf5a2bd4d15a60ae982b3a15bcf7bfa7a0187d307cc19d2 not found: ID does not exist" Sep 11 20:03:02.235748 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.235747 2546 scope.go:117] "RemoveContainer" containerID="4d7aa46e3dc629a5238ca22dc98949c1627827ca573e5c5fba3b700cd5d6098b" Sep 11 20:03:02.235910 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.235831 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 20:03:02.236039 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.236019 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d7aa46e3dc629a5238ca22dc98949c1627827ca573e5c5fba3b700cd5d6098b"} err="failed to get container status \"4d7aa46e3dc629a5238ca22dc98949c1627827ca573e5c5fba3b700cd5d6098b\": rpc error: code = NotFound desc = could not find container \"4d7aa46e3dc629a5238ca22dc98949c1627827ca573e5c5fba3b700cd5d6098b\": container with ID starting with 4d7aa46e3dc629a5238ca22dc98949c1627827ca573e5c5fba3b700cd5d6098b not found: ID does not exist" Sep 11 20:03:02.236104 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.236040 2546 scope.go:117] "RemoveContainer" containerID="1e26caaa2e13b8294f8c63134fac5d98860699d9d18175113d22919b02b72e81" Sep 11 20:03:02.236104 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.236096 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71e7ab2f-2146-48a4-a9a0-b6ac3de11385" containerName="init-config-reloader" Sep 11 20:03:02.236204 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.236108 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="71e7ab2f-2146-48a4-a9a0-b6ac3de11385" containerName="init-config-reloader" Sep 11 20:03:02.236204 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.236118 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71e7ab2f-2146-48a4-a9a0-b6ac3de11385" containerName="kube-rbac-proxy" Sep 11 20:03:02.236204 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.236124 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="71e7ab2f-2146-48a4-a9a0-b6ac3de11385" containerName="kube-rbac-proxy" Sep 11 20:03:02.236204 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.236140 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71e7ab2f-2146-48a4-a9a0-b6ac3de11385" containerName="prometheus" Sep 11 20:03:02.236204 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.236148 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="71e7ab2f-2146-48a4-a9a0-b6ac3de11385" containerName="prometheus" Sep 11 20:03:02.236204 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.236162 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71e7ab2f-2146-48a4-a9a0-b6ac3de11385" containerName="config-reloader" Sep 11 20:03:02.236204 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.236168 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="71e7ab2f-2146-48a4-a9a0-b6ac3de11385" containerName="config-reloader" Sep 11 20:03:02.236204 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.236175 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71e7ab2f-2146-48a4-a9a0-b6ac3de11385" containerName="thanos-sidecar" Sep 11 20:03:02.236204 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.236184 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="71e7ab2f-2146-48a4-a9a0-b6ac3de11385" containerName="thanos-sidecar" Sep 11 20:03:02.236204 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.236192 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71e7ab2f-2146-48a4-a9a0-b6ac3de11385" containerName="kube-rbac-proxy-web" Sep 11 20:03:02.236204 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.236200 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="71e7ab2f-2146-48a4-a9a0-b6ac3de11385" containerName="kube-rbac-proxy-web" Sep 11 20:03:02.236561 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.236214 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71e7ab2f-2146-48a4-a9a0-b6ac3de11385" containerName="kube-rbac-proxy-thanos" Sep 11 20:03:02.236561 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.236222 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="71e7ab2f-2146-48a4-a9a0-b6ac3de11385" containerName="kube-rbac-proxy-thanos" Sep 11 20:03:02.236561 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.236274 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="71e7ab2f-2146-48a4-a9a0-b6ac3de11385" containerName="kube-rbac-proxy-web" Sep 11 20:03:02.236561 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.236274 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e26caaa2e13b8294f8c63134fac5d98860699d9d18175113d22919b02b72e81"} err="failed to get container status \"1e26caaa2e13b8294f8c63134fac5d98860699d9d18175113d22919b02b72e81\": rpc error: code = NotFound desc = could not find container \"1e26caaa2e13b8294f8c63134fac5d98860699d9d18175113d22919b02b72e81\": container with ID starting with 1e26caaa2e13b8294f8c63134fac5d98860699d9d18175113d22919b02b72e81 not found: ID does not exist" Sep 11 20:03:02.236561 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.236294 2546 scope.go:117] "RemoveContainer" containerID="8be7262fa9233d57eaf437bb764aad9813b8595a143f36d393faaff8651e22a2" Sep 11 20:03:02.236561 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.236282 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="71e7ab2f-2146-48a4-a9a0-b6ac3de11385" containerName="prometheus" Sep 11 20:03:02.236561 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.236342 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="71e7ab2f-2146-48a4-a9a0-b6ac3de11385" containerName="config-reloader" Sep 11 20:03:02.236561 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.236350 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="71e7ab2f-2146-48a4-a9a0-b6ac3de11385" containerName="thanos-sidecar" Sep 11 20:03:02.236561 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.236356 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="71e7ab2f-2146-48a4-a9a0-b6ac3de11385" containerName="kube-rbac-proxy-thanos" Sep 11 20:03:02.236561 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.236362 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="71e7ab2f-2146-48a4-a9a0-b6ac3de11385" containerName="kube-rbac-proxy" Sep 11 20:03:02.236561 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.236558 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8be7262fa9233d57eaf437bb764aad9813b8595a143f36d393faaff8651e22a2"} err="failed to get container status \"8be7262fa9233d57eaf437bb764aad9813b8595a143f36d393faaff8651e22a2\": rpc error: code = NotFound desc = could not find container \"8be7262fa9233d57eaf437bb764aad9813b8595a143f36d393faaff8651e22a2\": container with ID starting with 8be7262fa9233d57eaf437bb764aad9813b8595a143f36d393faaff8651e22a2 not found: ID does not exist" Sep 11 20:03:02.236962 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.236592 2546 scope.go:117] "RemoveContainer" containerID="c6f7823a9b1c8f43ddc1bccbbe4e12f3bc445ae16a4f9836609282adb6b8b055" Sep 11 20:03:02.236962 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.236809 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6f7823a9b1c8f43ddc1bccbbe4e12f3bc445ae16a4f9836609282adb6b8b055"} err="failed to get container status \"c6f7823a9b1c8f43ddc1bccbbe4e12f3bc445ae16a4f9836609282adb6b8b055\": rpc error: code = NotFound desc = could not find container \"c6f7823a9b1c8f43ddc1bccbbe4e12f3bc445ae16a4f9836609282adb6b8b055\": container with ID starting with c6f7823a9b1c8f43ddc1bccbbe4e12f3bc445ae16a4f9836609282adb6b8b055 not found: ID does not exist" Sep 11 20:03:02.236962 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.236825 2546 scope.go:117] "RemoveContainer" containerID="67da36248892982f4627891345c94bbbdd9f80fdbffd1df19b33417829189d75" Sep 11 20:03:02.237086 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.237059 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67da36248892982f4627891345c94bbbdd9f80fdbffd1df19b33417829189d75"} err="failed to get container status \"67da36248892982f4627891345c94bbbdd9f80fdbffd1df19b33417829189d75\": rpc error: code = NotFound desc = could not find container \"67da36248892982f4627891345c94bbbdd9f80fdbffd1df19b33417829189d75\": container with ID starting with 67da36248892982f4627891345c94bbbdd9f80fdbffd1df19b33417829189d75 not found: ID does not exist" Sep 11 20:03:02.237086 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.237078 2546 scope.go:117] "RemoveContainer" containerID="850eec169eb8fb81640c4dbcef362bdd0da6665e52975d0df320f24d7584571b" Sep 11 20:03:02.237307 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.237291 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"850eec169eb8fb81640c4dbcef362bdd0da6665e52975d0df320f24d7584571b"} err="failed to get container status \"850eec169eb8fb81640c4dbcef362bdd0da6665e52975d0df320f24d7584571b\": rpc error: code = NotFound desc = could not find container \"850eec169eb8fb81640c4dbcef362bdd0da6665e52975d0df320f24d7584571b\": container with ID starting with 850eec169eb8fb81640c4dbcef362bdd0da6665e52975d0df320f24d7584571b not found: ID does not exist" Sep 11 20:03:02.237307 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.237306 2546 scope.go:117] "RemoveContainer" containerID="4487c4d66faf952bccf5a2bd4d15a60ae982b3a15bcf7bfa7a0187d307cc19d2" Sep 11 20:03:02.237546 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.237527 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4487c4d66faf952bccf5a2bd4d15a60ae982b3a15bcf7bfa7a0187d307cc19d2"} err="failed to get container status \"4487c4d66faf952bccf5a2bd4d15a60ae982b3a15bcf7bfa7a0187d307cc19d2\": rpc error: code = NotFound desc = could not find container \"4487c4d66faf952bccf5a2bd4d15a60ae982b3a15bcf7bfa7a0187d307cc19d2\": container with ID starting with 4487c4d66faf952bccf5a2bd4d15a60ae982b3a15bcf7bfa7a0187d307cc19d2 not found: ID does not exist" Sep 11 20:03:02.237546 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.237546 2546 scope.go:117] "RemoveContainer" containerID="4d7aa46e3dc629a5238ca22dc98949c1627827ca573e5c5fba3b700cd5d6098b" Sep 11 20:03:02.237807 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.237789 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d7aa46e3dc629a5238ca22dc98949c1627827ca573e5c5fba3b700cd5d6098b"} err="failed to get container status \"4d7aa46e3dc629a5238ca22dc98949c1627827ca573e5c5fba3b700cd5d6098b\": rpc error: code = NotFound desc = could not find container \"4d7aa46e3dc629a5238ca22dc98949c1627827ca573e5c5fba3b700cd5d6098b\": container with ID starting with 4d7aa46e3dc629a5238ca22dc98949c1627827ca573e5c5fba3b700cd5d6098b not found: ID does not exist" Sep 11 20:03:02.237850 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.237808 2546 scope.go:117] "RemoveContainer" containerID="1e26caaa2e13b8294f8c63134fac5d98860699d9d18175113d22919b02b72e81" Sep 11 20:03:02.238030 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.238014 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e26caaa2e13b8294f8c63134fac5d98860699d9d18175113d22919b02b72e81"} err="failed to get container status \"1e26caaa2e13b8294f8c63134fac5d98860699d9d18175113d22919b02b72e81\": rpc error: code = NotFound desc = could not find container \"1e26caaa2e13b8294f8c63134fac5d98860699d9d18175113d22919b02b72e81\": container with ID starting with 1e26caaa2e13b8294f8c63134fac5d98860699d9d18175113d22919b02b72e81 not found: ID does not exist" Sep 11 20:03:02.238075 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.238032 2546 scope.go:117] "RemoveContainer" containerID="8be7262fa9233d57eaf437bb764aad9813b8595a143f36d393faaff8651e22a2" Sep 11 20:03:02.238266 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.238248 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8be7262fa9233d57eaf437bb764aad9813b8595a143f36d393faaff8651e22a2"} err="failed to get container status \"8be7262fa9233d57eaf437bb764aad9813b8595a143f36d393faaff8651e22a2\": rpc error: code = NotFound desc = could not find container \"8be7262fa9233d57eaf437bb764aad9813b8595a143f36d393faaff8651e22a2\": container with ID starting with 8be7262fa9233d57eaf437bb764aad9813b8595a143f36d393faaff8651e22a2 not found: ID does not exist" Sep 11 20:03:02.238320 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.238266 2546 scope.go:117] "RemoveContainer" containerID="c6f7823a9b1c8f43ddc1bccbbe4e12f3bc445ae16a4f9836609282adb6b8b055" Sep 11 20:03:02.238532 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.238489 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6f7823a9b1c8f43ddc1bccbbe4e12f3bc445ae16a4f9836609282adb6b8b055"} err="failed to get container status \"c6f7823a9b1c8f43ddc1bccbbe4e12f3bc445ae16a4f9836609282adb6b8b055\": rpc error: code = NotFound desc = could not find container \"c6f7823a9b1c8f43ddc1bccbbe4e12f3bc445ae16a4f9836609282adb6b8b055\": container with ID starting with c6f7823a9b1c8f43ddc1bccbbe4e12f3bc445ae16a4f9836609282adb6b8b055 not found: ID does not exist" Sep 11 20:03:02.238532 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.238513 2546 scope.go:117] "RemoveContainer" containerID="67da36248892982f4627891345c94bbbdd9f80fdbffd1df19b33417829189d75" Sep 11 20:03:02.238890 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.238860 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67da36248892982f4627891345c94bbbdd9f80fdbffd1df19b33417829189d75"} err="failed to get container status \"67da36248892982f4627891345c94bbbdd9f80fdbffd1df19b33417829189d75\": rpc error: code = NotFound desc = could not find container \"67da36248892982f4627891345c94bbbdd9f80fdbffd1df19b33417829189d75\": container with ID starting with 67da36248892982f4627891345c94bbbdd9f80fdbffd1df19b33417829189d75 not found: ID does not exist" Sep 11 20:03:02.238956 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.238890 2546 scope.go:117] "RemoveContainer" containerID="850eec169eb8fb81640c4dbcef362bdd0da6665e52975d0df320f24d7584571b" Sep 11 20:03:02.239167 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.239148 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"850eec169eb8fb81640c4dbcef362bdd0da6665e52975d0df320f24d7584571b"} err="failed to get container status \"850eec169eb8fb81640c4dbcef362bdd0da6665e52975d0df320f24d7584571b\": rpc error: code = NotFound desc = could not find container \"850eec169eb8fb81640c4dbcef362bdd0da6665e52975d0df320f24d7584571b\": container with ID starting with 850eec169eb8fb81640c4dbcef362bdd0da6665e52975d0df320f24d7584571b not found: ID does not exist" Sep 11 20:03:02.239221 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.239167 2546 scope.go:117] "RemoveContainer" containerID="4487c4d66faf952bccf5a2bd4d15a60ae982b3a15bcf7bfa7a0187d307cc19d2" Sep 11 20:03:02.239407 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.239389 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4487c4d66faf952bccf5a2bd4d15a60ae982b3a15bcf7bfa7a0187d307cc19d2"} err="failed to get container status \"4487c4d66faf952bccf5a2bd4d15a60ae982b3a15bcf7bfa7a0187d307cc19d2\": rpc error: code = NotFound desc = could not find container \"4487c4d66faf952bccf5a2bd4d15a60ae982b3a15bcf7bfa7a0187d307cc19d2\": container with ID starting with 4487c4d66faf952bccf5a2bd4d15a60ae982b3a15bcf7bfa7a0187d307cc19d2 not found: ID does not exist" Sep 11 20:03:02.239450 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.239408 2546 scope.go:117] "RemoveContainer" containerID="4d7aa46e3dc629a5238ca22dc98949c1627827ca573e5c5fba3b700cd5d6098b" Sep 11 20:03:02.239638 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.239619 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d7aa46e3dc629a5238ca22dc98949c1627827ca573e5c5fba3b700cd5d6098b"} err="failed to get container status \"4d7aa46e3dc629a5238ca22dc98949c1627827ca573e5c5fba3b700cd5d6098b\": rpc error: code = NotFound desc = could not find container \"4d7aa46e3dc629a5238ca22dc98949c1627827ca573e5c5fba3b700cd5d6098b\": container with ID starting with 4d7aa46e3dc629a5238ca22dc98949c1627827ca573e5c5fba3b700cd5d6098b not found: ID does not exist" Sep 11 20:03:02.240074 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.240061 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:03:02.242192 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.242154 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 11 20:03:02.242192 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.242165 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 11 20:03:02.242326 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.242219 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 11 20:03:02.242326 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.242238 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 11 20:03:02.242326 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.242281 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-4mtrh\"" Sep 11 20:03:02.242518 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.242501 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 11 20:03:02.242673 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.242653 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 11 20:03:02.242908 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.242750 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 11 20:03:02.242908 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.242767 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 11 20:03:02.242908 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.242751 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 11 20:03:02.243133 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.243008 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 11 20:03:02.243133 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.243023 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-a8bpa4l5b72tl\"" Sep 11 20:03:02.243466 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.243446 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 11 20:03:02.243634 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.243558 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 11 20:03:02.243634 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.243559 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 11 20:03:02.248057 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.248037 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 11 20:03:02.251870 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.251852 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 11 20:03:02.263561 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.263535 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 20:03:02.339701 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.339674 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d87454d6-3a5a-4a45-b925-fe4f4bd5d697-config\") pod \"prometheus-k8s-0\" (UID: \"d87454d6-3a5a-4a45-b925-fe4f4bd5d697\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:03:02.339830 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.339713 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d87454d6-3a5a-4a45-b925-fe4f4bd5d697-config-out\") pod \"prometheus-k8s-0\" (UID: \"d87454d6-3a5a-4a45-b925-fe4f4bd5d697\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:03:02.339830 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.339788 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d87454d6-3a5a-4a45-b925-fe4f4bd5d697-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d87454d6-3a5a-4a45-b925-fe4f4bd5d697\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:03:02.339910 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.339836 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d87454d6-3a5a-4a45-b925-fe4f4bd5d697-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d87454d6-3a5a-4a45-b925-fe4f4bd5d697\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:03:02.339910 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.339855 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d87454d6-3a5a-4a45-b925-fe4f4bd5d697-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d87454d6-3a5a-4a45-b925-fe4f4bd5d697\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:03:02.339910 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.339879 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d87454d6-3a5a-4a45-b925-fe4f4bd5d697-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d87454d6-3a5a-4a45-b925-fe4f4bd5d697\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:03:02.340033 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.339935 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d87454d6-3a5a-4a45-b925-fe4f4bd5d697-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d87454d6-3a5a-4a45-b925-fe4f4bd5d697\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:03:02.340033 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.339951 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d87454d6-3a5a-4a45-b925-fe4f4bd5d697-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d87454d6-3a5a-4a45-b925-fe4f4bd5d697\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:03:02.340033 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.339971 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d87454d6-3a5a-4a45-b925-fe4f4bd5d697-web-config\") pod \"prometheus-k8s-0\" (UID: \"d87454d6-3a5a-4a45-b925-fe4f4bd5d697\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:03:02.340033 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.339994 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d87454d6-3a5a-4a45-b925-fe4f4bd5d697-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d87454d6-3a5a-4a45-b925-fe4f4bd5d697\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:03:02.340181 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.340076 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d87454d6-3a5a-4a45-b925-fe4f4bd5d697-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d87454d6-3a5a-4a45-b925-fe4f4bd5d697\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:03:02.340181 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.340097 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d87454d6-3a5a-4a45-b925-fe4f4bd5d697-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d87454d6-3a5a-4a45-b925-fe4f4bd5d697\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:03:02.340181 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.340122 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d87454d6-3a5a-4a45-b925-fe4f4bd5d697-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d87454d6-3a5a-4a45-b925-fe4f4bd5d697\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:03:02.340181 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.340146 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5fnn\" (UniqueName: \"kubernetes.io/projected/d87454d6-3a5a-4a45-b925-fe4f4bd5d697-kube-api-access-k5fnn\") pod \"prometheus-k8s-0\" (UID: \"d87454d6-3a5a-4a45-b925-fe4f4bd5d697\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:03:02.340181 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.340167 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d87454d6-3a5a-4a45-b925-fe4f4bd5d697-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d87454d6-3a5a-4a45-b925-fe4f4bd5d697\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:03:02.340404 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.340183 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d87454d6-3a5a-4a45-b925-fe4f4bd5d697-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d87454d6-3a5a-4a45-b925-fe4f4bd5d697\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:03:02.340404 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.340204 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d87454d6-3a5a-4a45-b925-fe4f4bd5d697-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d87454d6-3a5a-4a45-b925-fe4f4bd5d697\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:03:02.340404 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.340217 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d87454d6-3a5a-4a45-b925-fe4f4bd5d697-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d87454d6-3a5a-4a45-b925-fe4f4bd5d697\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:03:02.340404 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.340231 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d87454d6-3a5a-4a45-b925-fe4f4bd5d697-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d87454d6-3a5a-4a45-b925-fe4f4bd5d697\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:03:02.340404 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.340248 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d87454d6-3a5a-4a45-b925-fe4f4bd5d697-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d87454d6-3a5a-4a45-b925-fe4f4bd5d697\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:03:02.441093 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.441022 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d87454d6-3a5a-4a45-b925-fe4f4bd5d697-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d87454d6-3a5a-4a45-b925-fe4f4bd5d697\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:03:02.441093 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.441053 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d87454d6-3a5a-4a45-b925-fe4f4bd5d697-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d87454d6-3a5a-4a45-b925-fe4f4bd5d697\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:03:02.441093 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.441076 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d87454d6-3a5a-4a45-b925-fe4f4bd5d697-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d87454d6-3a5a-4a45-b925-fe4f4bd5d697\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:03:02.441329 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.441102 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d87454d6-3a5a-4a45-b925-fe4f4bd5d697-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d87454d6-3a5a-4a45-b925-fe4f4bd5d697\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:03:02.441329 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.441127 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d87454d6-3a5a-4a45-b925-fe4f4bd5d697-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d87454d6-3a5a-4a45-b925-fe4f4bd5d697\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:03:02.441329 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.441156 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d87454d6-3a5a-4a45-b925-fe4f4bd5d697-config\") pod \"prometheus-k8s-0\" (UID: \"d87454d6-3a5a-4a45-b925-fe4f4bd5d697\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:03:02.441329 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.441183 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d87454d6-3a5a-4a45-b925-fe4f4bd5d697-config-out\") pod \"prometheus-k8s-0\" (UID: \"d87454d6-3a5a-4a45-b925-fe4f4bd5d697\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:03:02.441329 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.441215 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d87454d6-3a5a-4a45-b925-fe4f4bd5d697-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d87454d6-3a5a-4a45-b925-fe4f4bd5d697\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:03:02.441329 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.441244 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d87454d6-3a5a-4a45-b925-fe4f4bd5d697-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d87454d6-3a5a-4a45-b925-fe4f4bd5d697\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:03:02.441329 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.441268 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d87454d6-3a5a-4a45-b925-fe4f4bd5d697-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d87454d6-3a5a-4a45-b925-fe4f4bd5d697\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:03:02.441329 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.441291 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d87454d6-3a5a-4a45-b925-fe4f4bd5d697-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d87454d6-3a5a-4a45-b925-fe4f4bd5d697\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:03:02.441733 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.441334 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d87454d6-3a5a-4a45-b925-fe4f4bd5d697-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d87454d6-3a5a-4a45-b925-fe4f4bd5d697\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:03:02.441733 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.441353 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d87454d6-3a5a-4a45-b925-fe4f4bd5d697-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d87454d6-3a5a-4a45-b925-fe4f4bd5d697\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:03:02.441733 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.441384 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d87454d6-3a5a-4a45-b925-fe4f4bd5d697-web-config\") pod \"prometheus-k8s-0\" (UID: \"d87454d6-3a5a-4a45-b925-fe4f4bd5d697\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:03:02.441733 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.441408 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d87454d6-3a5a-4a45-b925-fe4f4bd5d697-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d87454d6-3a5a-4a45-b925-fe4f4bd5d697\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:03:02.441733 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.441443 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d87454d6-3a5a-4a45-b925-fe4f4bd5d697-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d87454d6-3a5a-4a45-b925-fe4f4bd5d697\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:03:02.441733 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.441470 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d87454d6-3a5a-4a45-b925-fe4f4bd5d697-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d87454d6-3a5a-4a45-b925-fe4f4bd5d697\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:03:02.441733 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.441498 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d87454d6-3a5a-4a45-b925-fe4f4bd5d697-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d87454d6-3a5a-4a45-b925-fe4f4bd5d697\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:03:02.441733 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.441532 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5fnn\" (UniqueName: \"kubernetes.io/projected/d87454d6-3a5a-4a45-b925-fe4f4bd5d697-kube-api-access-k5fnn\") pod \"prometheus-k8s-0\" (UID: \"d87454d6-3a5a-4a45-b925-fe4f4bd5d697\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:03:02.441733 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.441566 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d87454d6-3a5a-4a45-b925-fe4f4bd5d697-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d87454d6-3a5a-4a45-b925-fe4f4bd5d697\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:03:02.442176 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.441897 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d87454d6-3a5a-4a45-b925-fe4f4bd5d697-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d87454d6-3a5a-4a45-b925-fe4f4bd5d697\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:03:02.442176 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.442091 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d87454d6-3a5a-4a45-b925-fe4f4bd5d697-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d87454d6-3a5a-4a45-b925-fe4f4bd5d697\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:03:02.446821 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.443337 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d87454d6-3a5a-4a45-b925-fe4f4bd5d697-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d87454d6-3a5a-4a45-b925-fe4f4bd5d697\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:03:02.446821 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.443374 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d87454d6-3a5a-4a45-b925-fe4f4bd5d697-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d87454d6-3a5a-4a45-b925-fe4f4bd5d697\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:03:02.446821 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.444436 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d87454d6-3a5a-4a45-b925-fe4f4bd5d697-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d87454d6-3a5a-4a45-b925-fe4f4bd5d697\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:03:02.446821 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.444569 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d87454d6-3a5a-4a45-b925-fe4f4bd5d697-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d87454d6-3a5a-4a45-b925-fe4f4bd5d697\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:03:02.446821 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.444844 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d87454d6-3a5a-4a45-b925-fe4f4bd5d697-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d87454d6-3a5a-4a45-b925-fe4f4bd5d697\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:03:02.446821 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.445003 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d87454d6-3a5a-4a45-b925-fe4f4bd5d697-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d87454d6-3a5a-4a45-b925-fe4f4bd5d697\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:03:02.446821 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.445233 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/d87454d6-3a5a-4a45-b925-fe4f4bd5d697-config\") pod \"prometheus-k8s-0\" (UID: \"d87454d6-3a5a-4a45-b925-fe4f4bd5d697\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:03:02.446821 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.445424 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d87454d6-3a5a-4a45-b925-fe4f4bd5d697-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d87454d6-3a5a-4a45-b925-fe4f4bd5d697\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:03:02.446821 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.445954 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d87454d6-3a5a-4a45-b925-fe4f4bd5d697-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d87454d6-3a5a-4a45-b925-fe4f4bd5d697\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:03:02.446821 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.446013 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d87454d6-3a5a-4a45-b925-fe4f4bd5d697-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d87454d6-3a5a-4a45-b925-fe4f4bd5d697\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:03:02.446821 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.446428 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d87454d6-3a5a-4a45-b925-fe4f4bd5d697-config-out\") pod \"prometheus-k8s-0\" (UID: \"d87454d6-3a5a-4a45-b925-fe4f4bd5d697\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:03:02.446821 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.446504 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d87454d6-3a5a-4a45-b925-fe4f4bd5d697-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d87454d6-3a5a-4a45-b925-fe4f4bd5d697\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:03:02.446821 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.446551 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d87454d6-3a5a-4a45-b925-fe4f4bd5d697-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d87454d6-3a5a-4a45-b925-fe4f4bd5d697\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:03:02.447555 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.446861 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d87454d6-3a5a-4a45-b925-fe4f4bd5d697-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d87454d6-3a5a-4a45-b925-fe4f4bd5d697\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:03:02.447555 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.447203 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d87454d6-3a5a-4a45-b925-fe4f4bd5d697-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d87454d6-3a5a-4a45-b925-fe4f4bd5d697\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:03:02.447555 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.447517 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d87454d6-3a5a-4a45-b925-fe4f4bd5d697-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d87454d6-3a5a-4a45-b925-fe4f4bd5d697\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:03:02.447972 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.447957 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d87454d6-3a5a-4a45-b925-fe4f4bd5d697-web-config\") pod \"prometheus-k8s-0\" (UID: \"d87454d6-3a5a-4a45-b925-fe4f4bd5d697\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:03:02.450507 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.450480 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71e7ab2f-2146-48a4-a9a0-b6ac3de11385" path="/var/lib/kubelet/pods/71e7ab2f-2146-48a4-a9a0-b6ac3de11385/volumes" Sep 11 20:03:02.451464 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.451444 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5fnn\" (UniqueName: \"kubernetes.io/projected/d87454d6-3a5a-4a45-b925-fe4f4bd5d697-kube-api-access-k5fnn\") pod \"prometheus-k8s-0\" (UID: \"d87454d6-3a5a-4a45-b925-fe4f4bd5d697\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:03:02.550329 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.550290 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:03:02.835905 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:02.835875 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 20:03:02.837691 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:03:02.837663 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd87454d6_3a5a_4a45_b925_fe4f4bd5d697.slice/crio-4950ffee68051576938825e0f02df5d6243101ada7cf670f4fa3c4a2ac596370 WatchSource:0}: Error finding container 4950ffee68051576938825e0f02df5d6243101ada7cf670f4fa3c4a2ac596370: Status 404 returned error can't find the container with id 4950ffee68051576938825e0f02df5d6243101ada7cf670f4fa3c4a2ac596370 Sep 11 20:03:03.189483 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:03.189451 2546 generic.go:356] "Generic (PLEG): container finished" podID="d87454d6-3a5a-4a45-b925-fe4f4bd5d697" containerID="1dc5dd6067d45040c3650135b77aa87d08e7f15757c17361ef01d39fd8f3ef6d" exitCode=0 Sep 11 20:03:03.189834 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:03.189508 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d87454d6-3a5a-4a45-b925-fe4f4bd5d697","Type":"ContainerDied","Data":"1dc5dd6067d45040c3650135b77aa87d08e7f15757c17361ef01d39fd8f3ef6d"} Sep 11 20:03:03.189834 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:03.189556 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d87454d6-3a5a-4a45-b925-fe4f4bd5d697","Type":"ContainerStarted","Data":"4950ffee68051576938825e0f02df5d6243101ada7cf670f4fa3c4a2ac596370"} Sep 11 20:03:04.195633 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:04.195604 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-fc5d4f5c9-bksvl" event={"ID":"e9849d8f-4428-4c98-9aeb-f02cc00775d3","Type":"ContainerStarted","Data":"f27ff7b6afb449ac04f07433b6102b46ac2b48202d4d4aed23efa2b0c22479bb"} Sep 11 20:03:04.196023 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:04.195661 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-fc5d4f5c9-bksvl" event={"ID":"e9849d8f-4428-4c98-9aeb-f02cc00775d3","Type":"ContainerStarted","Data":"3dfde8eb3bbed052bcaa87fe6b2895f134d965bacb5cf8ec53f219de9c333d4e"} Sep 11 20:03:04.196023 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:04.195691 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-fc5d4f5c9-bksvl" event={"ID":"e9849d8f-4428-4c98-9aeb-f02cc00775d3","Type":"ContainerStarted","Data":"18458835b1b4293235abf58765fd456768485cfcfd098ed8d2fbb935db5c4cf5"} Sep 11 20:03:04.199079 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:04.199050 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d87454d6-3a5a-4a45-b925-fe4f4bd5d697","Type":"ContainerStarted","Data":"2b9032aff80299763b01a76d19043edc70e3ed400bfac58e76c626239efb96c5"} Sep 11 20:03:04.199197 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:04.199121 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d87454d6-3a5a-4a45-b925-fe4f4bd5d697","Type":"ContainerStarted","Data":"6d67b3a5751a1160a00e8ba1117c0d120c4419b6f1d26201b8321f5878d419c9"} Sep 11 20:03:04.199197 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:04.199171 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d87454d6-3a5a-4a45-b925-fe4f4bd5d697","Type":"ContainerStarted","Data":"e89b05d97d1a13fbbac291a006848eb11b54a9394d0ad973664a7fd3f1b8b0b2"} Sep 11 20:03:04.199326 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:04.199213 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d87454d6-3a5a-4a45-b925-fe4f4bd5d697","Type":"ContainerStarted","Data":"effc5fd3001920ff4f8f0e8789cc0dc4622a06b1f7c8d10ed071bee822cbd71b"} Sep 11 20:03:04.199326 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:04.199258 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d87454d6-3a5a-4a45-b925-fe4f4bd5d697","Type":"ContainerStarted","Data":"0801408cf6fa3c6058a5637c4cc07fe6688d267a5b4c8708fdb11ffd7c8247f9"} Sep 11 20:03:04.199326 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:04.199296 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d87454d6-3a5a-4a45-b925-fe4f4bd5d697","Type":"ContainerStarted","Data":"4b88e35bed7084c18a3440c2c4704c80c807b4ef0339c835e7753e6bc9b67976"} Sep 11 20:03:04.253286 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:04.253237 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.233085475 podStartE2EDuration="2.253220058s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.134583ms" isStatefulPod=false podCreationTimestamp="2026-09-11 20:03:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:03:04.2514078 +0000 UTC m=+154.299458187" watchObservedRunningTime="2026-09-11 20:03:04.253220058 +0000 UTC m=+154.301270403" Sep 11 20:03:04.253476 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:04.253447 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-fc5d4f5c9-bksvl" podStartSLOduration=1.135561181 podStartE2EDuration="3.253439346s" totalImagesPullingTime="2.117878165s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:03:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:03:04.221732911 +0000 UTC m=+154.269783257" watchObservedRunningTime="2026-09-11 20:03:04.253439346 +0000 UTC m=+154.301489692" Sep 11 20:03:04.361617 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:04.361589 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:03:04.419550 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:04.419524 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:03:07.550664 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:07.550626 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:03:15.239401 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:15.239364 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66875d4f5b-gt7sc"] Sep 11 20:03:15.344951 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:15.344921 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:03:15.449096 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:15.449063 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:03:15.545875 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:15.545812 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:03:40.263763 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:40.263690 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-66875d4f5b-gt7sc" podUID="8dc9b50e-1594-425a-aa20-8f355bdf6d65" containerName="console" containerID="cri-o://22bea90f988f3958b0110a81e95149e6bbd8662969b8ecea0d9f32d40fee4427" gracePeriod=15 Sep 11 20:03:40.503082 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:40.503060 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66875d4f5b-gt7sc" Sep 11 20:03:40.635860 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:40.635825 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8dc9b50e-1594-425a-aa20-8f355bdf6d65-oauth-serving-cert\") pod \"8dc9b50e-1594-425a-aa20-8f355bdf6d65\" (UID: \"8dc9b50e-1594-425a-aa20-8f355bdf6d65\") " Sep 11 20:03:40.635860 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:40.635864 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8dc9b50e-1594-425a-aa20-8f355bdf6d65-service-ca\") pod \"8dc9b50e-1594-425a-aa20-8f355bdf6d65\" (UID: \"8dc9b50e-1594-425a-aa20-8f355bdf6d65\") " Sep 11 20:03:40.636072 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:40.635888 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-svg6m\" (UniqueName: \"kubernetes.io/projected/8dc9b50e-1594-425a-aa20-8f355bdf6d65-kube-api-access-svg6m\") pod \"8dc9b50e-1594-425a-aa20-8f355bdf6d65\" (UID: \"8dc9b50e-1594-425a-aa20-8f355bdf6d65\") " Sep 11 20:03:40.636072 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:40.635935 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8dc9b50e-1594-425a-aa20-8f355bdf6d65-console-oauth-config\") pod \"8dc9b50e-1594-425a-aa20-8f355bdf6d65\" (UID: \"8dc9b50e-1594-425a-aa20-8f355bdf6d65\") " Sep 11 20:03:40.636072 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:40.635995 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8dc9b50e-1594-425a-aa20-8f355bdf6d65-console-config\") pod \"8dc9b50e-1594-425a-aa20-8f355bdf6d65\" (UID: \"8dc9b50e-1594-425a-aa20-8f355bdf6d65\") " Sep 11 20:03:40.636072 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:40.636023 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8dc9b50e-1594-425a-aa20-8f355bdf6d65-trusted-ca-bundle\") pod \"8dc9b50e-1594-425a-aa20-8f355bdf6d65\" (UID: \"8dc9b50e-1594-425a-aa20-8f355bdf6d65\") " Sep 11 20:03:40.636072 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:40.636056 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8dc9b50e-1594-425a-aa20-8f355bdf6d65-console-serving-cert\") pod \"8dc9b50e-1594-425a-aa20-8f355bdf6d65\" (UID: \"8dc9b50e-1594-425a-aa20-8f355bdf6d65\") " Sep 11 20:03:40.636321 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:40.636260 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dc9b50e-1594-425a-aa20-8f355bdf6d65-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8dc9b50e-1594-425a-aa20-8f355bdf6d65" (UID: "8dc9b50e-1594-425a-aa20-8f355bdf6d65"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:03:40.636402 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:40.636277 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dc9b50e-1594-425a-aa20-8f355bdf6d65-service-ca" (OuterVolumeSpecName: "service-ca") pod "8dc9b50e-1594-425a-aa20-8f355bdf6d65" (UID: "8dc9b50e-1594-425a-aa20-8f355bdf6d65"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:03:40.636515 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:40.636483 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dc9b50e-1594-425a-aa20-8f355bdf6d65-console-config" (OuterVolumeSpecName: "console-config") pod "8dc9b50e-1594-425a-aa20-8f355bdf6d65" (UID: "8dc9b50e-1594-425a-aa20-8f355bdf6d65"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:03:40.636515 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:40.636502 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dc9b50e-1594-425a-aa20-8f355bdf6d65-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "8dc9b50e-1594-425a-aa20-8f355bdf6d65" (UID: "8dc9b50e-1594-425a-aa20-8f355bdf6d65"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:03:40.638036 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:40.638007 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dc9b50e-1594-425a-aa20-8f355bdf6d65-kube-api-access-svg6m" (OuterVolumeSpecName: "kube-api-access-svg6m") pod "8dc9b50e-1594-425a-aa20-8f355bdf6d65" (UID: "8dc9b50e-1594-425a-aa20-8f355bdf6d65"). InnerVolumeSpecName "kube-api-access-svg6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:03:40.638165 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:40.638144 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dc9b50e-1594-425a-aa20-8f355bdf6d65-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8dc9b50e-1594-425a-aa20-8f355bdf6d65" (UID: "8dc9b50e-1594-425a-aa20-8f355bdf6d65"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 20:03:40.638212 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:40.638172 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dc9b50e-1594-425a-aa20-8f355bdf6d65-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8dc9b50e-1594-425a-aa20-8f355bdf6d65" (UID: "8dc9b50e-1594-425a-aa20-8f355bdf6d65"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 20:03:40.737297 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:40.737261 2546 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8dc9b50e-1594-425a-aa20-8f355bdf6d65-oauth-serving-cert\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:03:40.737297 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:40.737294 2546 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8dc9b50e-1594-425a-aa20-8f355bdf6d65-service-ca\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:03:40.737297 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:40.737306 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svg6m\" (UniqueName: \"kubernetes.io/projected/8dc9b50e-1594-425a-aa20-8f355bdf6d65-kube-api-access-svg6m\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:03:40.737475 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:40.737316 2546 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8dc9b50e-1594-425a-aa20-8f355bdf6d65-console-oauth-config\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:03:40.737475 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:40.737324 2546 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8dc9b50e-1594-425a-aa20-8f355bdf6d65-console-config\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:03:40.737475 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:40.737332 2546 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8dc9b50e-1594-425a-aa20-8f355bdf6d65-trusted-ca-bundle\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:03:40.737475 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:40.737340 2546 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8dc9b50e-1594-425a-aa20-8f355bdf6d65-console-serving-cert\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:03:41.344769 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:41.344730 2546 generic.go:356] "Generic (PLEG): container finished" podID="8dc9b50e-1594-425a-aa20-8f355bdf6d65" containerID="22bea90f988f3958b0110a81e95149e6bbd8662969b8ecea0d9f32d40fee4427" exitCode=0 Sep 11 20:03:41.345158 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:41.344799 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66875d4f5b-gt7sc" Sep 11 20:03:41.345158 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:41.344820 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66875d4f5b-gt7sc" event={"ID":"8dc9b50e-1594-425a-aa20-8f355bdf6d65","Type":"ContainerDied","Data":"22bea90f988f3958b0110a81e95149e6bbd8662969b8ecea0d9f32d40fee4427"} Sep 11 20:03:41.345158 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:41.344905 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66875d4f5b-gt7sc" event={"ID":"8dc9b50e-1594-425a-aa20-8f355bdf6d65","Type":"ContainerDied","Data":"877674da9e7846f09647273c04a3925a9280a154d137f7ecae1ed1da4b01700d"} Sep 11 20:03:41.345158 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:41.344927 2546 scope.go:117] "RemoveContainer" containerID="22bea90f988f3958b0110a81e95149e6bbd8662969b8ecea0d9f32d40fee4427" Sep 11 20:03:41.353324 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:41.353309 2546 scope.go:117] "RemoveContainer" containerID="22bea90f988f3958b0110a81e95149e6bbd8662969b8ecea0d9f32d40fee4427" Sep 11 20:03:41.353607 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:03:41.353568 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22bea90f988f3958b0110a81e95149e6bbd8662969b8ecea0d9f32d40fee4427\": container with ID starting with 22bea90f988f3958b0110a81e95149e6bbd8662969b8ecea0d9f32d40fee4427 not found: ID does not exist" containerID="22bea90f988f3958b0110a81e95149e6bbd8662969b8ecea0d9f32d40fee4427" Sep 11 20:03:41.353674 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:41.353620 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22bea90f988f3958b0110a81e95149e6bbd8662969b8ecea0d9f32d40fee4427"} err="failed to get container status \"22bea90f988f3958b0110a81e95149e6bbd8662969b8ecea0d9f32d40fee4427\": rpc error: code = NotFound desc = could not find container \"22bea90f988f3958b0110a81e95149e6bbd8662969b8ecea0d9f32d40fee4427\": container with ID starting with 22bea90f988f3958b0110a81e95149e6bbd8662969b8ecea0d9f32d40fee4427 not found: ID does not exist" Sep 11 20:03:41.364105 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:41.364087 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66875d4f5b-gt7sc"] Sep 11 20:03:41.367695 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:41.367678 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-66875d4f5b-gt7sc"] Sep 11 20:03:41.681302 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:41.681228 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:03:42.448949 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:03:42.448918 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8dc9b50e-1594-425a-aa20-8f355bdf6d65" path="/var/lib/kubelet/pods/8dc9b50e-1594-425a-aa20-8f355bdf6d65/volumes" Sep 11 20:04:02.551337 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:04:02.551296 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:04:02.566494 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:04:02.566467 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:04:03.443672 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:04:03.443645 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 20:04:11.216767 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:04:11.216730 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:04:11.269501 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:04:11.269474 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:04:15.167016 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:04:15.166980 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:04:15.222734 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:04:15.222708 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:04:15.315750 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:04:15.315726 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:04:15.371914 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:04:15.371886 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:04:15.913164 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:04:15.913120 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:04:15.966152 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:04:15.966129 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:04:23.171447 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:04:23.171412 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:04:23.229823 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:04:23.229798 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:04:24.562338 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:04:24.562305 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:04:24.623963 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:04:24.623924 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:04:25.602594 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:04:25.602549 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:04:25.654794 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:04:25.654765 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:04:25.713141 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:04:25.713116 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:04:29.079540 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:04:29.079497 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:04:29.180454 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:04:29.180427 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:04:52.516764 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:04:52.516732 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:04:52.614867 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:04:52.614828 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:05:10.236996 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:05:10.236962 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:05:10.338255 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:05:10.338228 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:05:19.589145 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:05:19.589105 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:05:21.008271 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:05:21.008225 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:05:23.392746 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:05:23.392714 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:05:23.449627 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:05:23.449571 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:05:24.604170 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:05:24.604118 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:05:24.713175 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:05:24.713144 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:05:25.083256 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:05:25.083160 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:05:25.182902 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:05:25.182873 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:05:26.732621 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:05:26.732591 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:05:27.146264 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:05:27.146223 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:05:30.179664 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:05:30.179632 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:05:30.240907 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:05:30.240871 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:05:30.372456 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:05:30.372428 2546 kubelet.go:1648] "Image garbage collection succeeded" Sep 11 20:05:34.747610 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:05:34.747564 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:05:36.315834 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:05:36.315803 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-jbzkq"] Sep 11 20:05:36.316188 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:05:36.316112 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dc9b50e-1594-425a-aa20-8f355bdf6d65" containerName="console" Sep 11 20:05:36.316188 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:05:36.316122 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dc9b50e-1594-425a-aa20-8f355bdf6d65" containerName="console" Sep 11 20:05:36.316188 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:05:36.316171 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="8dc9b50e-1594-425a-aa20-8f355bdf6d65" containerName="console" Sep 11 20:05:36.318898 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:05:36.318883 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-jbzkq" Sep 11 20:05:36.320881 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:05:36.320853 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 11 20:05:36.321099 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:05:36.321085 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-hd6lk\"" Sep 11 20:05:36.321471 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:05:36.321459 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 11 20:05:36.330564 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:05:36.330545 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-jbzkq"] Sep 11 20:05:36.406468 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:05:36.406441 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l46nk\" (UniqueName: \"kubernetes.io/projected/3bf1a3a6-cfd0-4bd3-931c-3251d1dd99a2-kube-api-access-l46nk\") pod \"cert-manager-55b4d954d5-jbzkq\" (UID: \"3bf1a3a6-cfd0-4bd3-931c-3251d1dd99a2\") " pod="cert-manager/cert-manager-55b4d954d5-jbzkq" Sep 11 20:05:36.406624 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:05:36.406475 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3bf1a3a6-cfd0-4bd3-931c-3251d1dd99a2-bound-sa-token\") pod \"cert-manager-55b4d954d5-jbzkq\" (UID: \"3bf1a3a6-cfd0-4bd3-931c-3251d1dd99a2\") " pod="cert-manager/cert-manager-55b4d954d5-jbzkq" Sep 11 20:05:36.507551 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:05:36.507513 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l46nk\" (UniqueName: \"kubernetes.io/projected/3bf1a3a6-cfd0-4bd3-931c-3251d1dd99a2-kube-api-access-l46nk\") pod \"cert-manager-55b4d954d5-jbzkq\" (UID: \"3bf1a3a6-cfd0-4bd3-931c-3251d1dd99a2\") " pod="cert-manager/cert-manager-55b4d954d5-jbzkq" Sep 11 20:05:36.507551 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:05:36.507558 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3bf1a3a6-cfd0-4bd3-931c-3251d1dd99a2-bound-sa-token\") pod \"cert-manager-55b4d954d5-jbzkq\" (UID: \"3bf1a3a6-cfd0-4bd3-931c-3251d1dd99a2\") " pod="cert-manager/cert-manager-55b4d954d5-jbzkq" Sep 11 20:05:36.515830 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:05:36.515809 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3bf1a3a6-cfd0-4bd3-931c-3251d1dd99a2-bound-sa-token\") pod \"cert-manager-55b4d954d5-jbzkq\" (UID: \"3bf1a3a6-cfd0-4bd3-931c-3251d1dd99a2\") " pod="cert-manager/cert-manager-55b4d954d5-jbzkq" Sep 11 20:05:36.515935 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:05:36.515881 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l46nk\" (UniqueName: \"kubernetes.io/projected/3bf1a3a6-cfd0-4bd3-931c-3251d1dd99a2-kube-api-access-l46nk\") pod \"cert-manager-55b4d954d5-jbzkq\" (UID: \"3bf1a3a6-cfd0-4bd3-931c-3251d1dd99a2\") " pod="cert-manager/cert-manager-55b4d954d5-jbzkq" Sep 11 20:05:36.645151 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:05:36.645121 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-jbzkq" Sep 11 20:05:36.915394 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:05:36.915312 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-jbzkq"] Sep 11 20:05:36.920151 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:05:36.920122 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3bf1a3a6_cfd0_4bd3_931c_3251d1dd99a2.slice/crio-86c3e5300ef8aae8f02f45a82c5cdeeb2e168950574af6824274aae1bff48221 WatchSource:0}: Error finding container 86c3e5300ef8aae8f02f45a82c5cdeeb2e168950574af6824274aae1bff48221: Status 404 returned error can't find the container with id 86c3e5300ef8aae8f02f45a82c5cdeeb2e168950574af6824274aae1bff48221 Sep 11 20:05:36.922038 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:05:36.922016 2546 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 20:05:37.781092 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:05:37.781046 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-jbzkq" event={"ID":"3bf1a3a6-cfd0-4bd3-931c-3251d1dd99a2","Type":"ContainerStarted","Data":"86c3e5300ef8aae8f02f45a82c5cdeeb2e168950574af6824274aae1bff48221"} Sep 11 20:05:40.794874 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:05:40.794841 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-jbzkq" event={"ID":"3bf1a3a6-cfd0-4bd3-931c-3251d1dd99a2","Type":"ContainerStarted","Data":"7bc10c335c3791eb53276d1192b149170bb41ac740f0ab20ff90f063098b5645"} Sep 11 20:05:40.819156 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:05:40.819111 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-jbzkq" podStartSLOduration=1.664028809 podStartE2EDuration="4.819095768s" totalImagesPullingTime="3.155066959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:05:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:05:40.817590457 +0000 UTC m=+310.865640793" watchObservedRunningTime="2026-09-11 20:05:40.819095768 +0000 UTC m=+310.867146111" Sep 11 20:05:41.075146 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:05:41.075075 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:05:41.164996 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:05:41.164960 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:05:42.690328 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:05:42.690296 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5t8mh"] Sep 11 20:05:42.693515 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:05:42.693499 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5t8mh" Sep 11 20:05:42.697273 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:05:42.697254 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 11 20:05:42.697366 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:05:42.697257 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 11 20:05:42.697879 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:05:42.697865 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-fqsz9\"" Sep 11 20:05:42.703875 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:05:42.703855 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5t8mh"] Sep 11 20:05:42.863179 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:05:42.863144 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/26729964-bf2e-4b51-9680-9efbfb654783-tmp\") pod \"openshift-lws-operator-fd8ccff4c-5t8mh\" (UID: \"26729964-bf2e-4b51-9680-9efbfb654783\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5t8mh" Sep 11 20:05:42.863344 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:05:42.863184 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tlk4\" (UniqueName: \"kubernetes.io/projected/26729964-bf2e-4b51-9680-9efbfb654783-kube-api-access-9tlk4\") pod \"openshift-lws-operator-fd8ccff4c-5t8mh\" (UID: \"26729964-bf2e-4b51-9680-9efbfb654783\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5t8mh" Sep 11 20:05:42.881439 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:05:42.881412 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:05:42.964393 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:05:42.964336 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tlk4\" (UniqueName: \"kubernetes.io/projected/26729964-bf2e-4b51-9680-9efbfb654783-kube-api-access-9tlk4\") pod \"openshift-lws-operator-fd8ccff4c-5t8mh\" (UID: \"26729964-bf2e-4b51-9680-9efbfb654783\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5t8mh" Sep 11 20:05:42.964490 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:05:42.964417 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/26729964-bf2e-4b51-9680-9efbfb654783-tmp\") pod \"openshift-lws-operator-fd8ccff4c-5t8mh\" (UID: \"26729964-bf2e-4b51-9680-9efbfb654783\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5t8mh" Sep 11 20:05:42.964761 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:05:42.964745 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/26729964-bf2e-4b51-9680-9efbfb654783-tmp\") pod \"openshift-lws-operator-fd8ccff4c-5t8mh\" (UID: \"26729964-bf2e-4b51-9680-9efbfb654783\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5t8mh" Sep 11 20:05:42.972210 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:05:42.972189 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tlk4\" (UniqueName: \"kubernetes.io/projected/26729964-bf2e-4b51-9680-9efbfb654783-kube-api-access-9tlk4\") pod \"openshift-lws-operator-fd8ccff4c-5t8mh\" (UID: \"26729964-bf2e-4b51-9680-9efbfb654783\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5t8mh" Sep 11 20:05:43.002023 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:05:43.002003 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5t8mh" Sep 11 20:05:43.272331 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:05:43.272261 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5t8mh"] Sep 11 20:05:43.275133 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:05:43.275109 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod26729964_bf2e_4b51_9680_9efbfb654783.slice/crio-2486d3265bba797e5fd853d71a37736c18fc0bbcbdbbb405450d05f9b1bffab3 WatchSource:0}: Error finding container 2486d3265bba797e5fd853d71a37736c18fc0bbcbdbbb405450d05f9b1bffab3: Status 404 returned error can't find the container with id 2486d3265bba797e5fd853d71a37736c18fc0bbcbdbbb405450d05f9b1bffab3 Sep 11 20:05:43.809962 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:05:43.809928 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5t8mh" event={"ID":"26729964-bf2e-4b51-9680-9efbfb654783","Type":"ContainerStarted","Data":"2486d3265bba797e5fd853d71a37736c18fc0bbcbdbbb405450d05f9b1bffab3"} Sep 11 20:05:46.825898 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:05:46.825865 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5t8mh" event={"ID":"26729964-bf2e-4b51-9680-9efbfb654783","Type":"ContainerStarted","Data":"da862eeb035e113992b5e2cede8fa1b458c6601d613a0db9a97baafa9d606702"} Sep 11 20:05:46.843137 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:05:46.843085 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5t8mh" podStartSLOduration=2.187636318 podStartE2EDuration="4.84306834s" totalImagesPullingTime="2.655432022s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:05:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:05:46.841620446 +0000 UTC m=+316.889670789" watchObservedRunningTime="2026-09-11 20:05:46.84306834 +0000 UTC m=+316.891118683" Sep 11 20:05:47.551420 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:05:47.551379 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:05:47.647854 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:05:47.647826 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:05:49.768249 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:05:49.768221 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:05:58.057203 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:05:58.057165 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:05:58.121637 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:05:58.121604 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:00.902045 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:00.902016 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:01.007789 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:01.007760 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:01.522171 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:01.522138 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:01.763711 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:01.763681 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:01.858187 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:01.858155 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:01.951930 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:01.951902 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5fdf48ff49-xthdd"] Sep 11 20:06:01.958786 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:01.958763 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5fdf48ff49-xthdd" Sep 11 20:06:01.961015 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:01.960997 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 11 20:06:01.961633 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:01.961617 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 11 20:06:01.961721 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:01.961703 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-jwddf\"" Sep 11 20:06:01.962149 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:01.962130 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 11 20:06:01.969894 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:01.969877 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 11 20:06:01.981805 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:01.981785 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5fdf48ff49-xthdd"] Sep 11 20:06:02.012008 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:02.011981 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/be81dfc2-533a-441c-950c-83a3c63a4c6c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5fdf48ff49-xthdd\" (UID: \"be81dfc2-533a-441c-950c-83a3c63a4c6c\") " pod="opendatahub/opendatahub-operator-controller-manager-5fdf48ff49-xthdd" Sep 11 20:06:02.012114 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:02.012026 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/be81dfc2-533a-441c-950c-83a3c63a4c6c-manifests\") pod \"opendatahub-operator-controller-manager-5fdf48ff49-xthdd\" (UID: \"be81dfc2-533a-441c-950c-83a3c63a4c6c\") " pod="opendatahub/opendatahub-operator-controller-manager-5fdf48ff49-xthdd" Sep 11 20:06:02.012114 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:02.012056 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/be81dfc2-533a-441c-950c-83a3c63a4c6c-tmp\") pod \"opendatahub-operator-controller-manager-5fdf48ff49-xthdd\" (UID: \"be81dfc2-533a-441c-950c-83a3c63a4c6c\") " pod="opendatahub/opendatahub-operator-controller-manager-5fdf48ff49-xthdd" Sep 11 20:06:02.012184 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:02.012115 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/be81dfc2-533a-441c-950c-83a3c63a4c6c-webhook-cert\") pod \"opendatahub-operator-controller-manager-5fdf48ff49-xthdd\" (UID: \"be81dfc2-533a-441c-950c-83a3c63a4c6c\") " pod="opendatahub/opendatahub-operator-controller-manager-5fdf48ff49-xthdd" Sep 11 20:06:02.012184 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:02.012143 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5fkw\" (UniqueName: \"kubernetes.io/projected/be81dfc2-533a-441c-950c-83a3c63a4c6c-kube-api-access-m5fkw\") pod \"opendatahub-operator-controller-manager-5fdf48ff49-xthdd\" (UID: \"be81dfc2-533a-441c-950c-83a3c63a4c6c\") " pod="opendatahub/opendatahub-operator-controller-manager-5fdf48ff49-xthdd" Sep 11 20:06:02.113039 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:02.112964 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/be81dfc2-533a-441c-950c-83a3c63a4c6c-manifests\") pod \"opendatahub-operator-controller-manager-5fdf48ff49-xthdd\" (UID: \"be81dfc2-533a-441c-950c-83a3c63a4c6c\") " pod="opendatahub/opendatahub-operator-controller-manager-5fdf48ff49-xthdd" Sep 11 20:06:02.113039 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:02.113002 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/be81dfc2-533a-441c-950c-83a3c63a4c6c-tmp\") pod \"opendatahub-operator-controller-manager-5fdf48ff49-xthdd\" (UID: \"be81dfc2-533a-441c-950c-83a3c63a4c6c\") " pod="opendatahub/opendatahub-operator-controller-manager-5fdf48ff49-xthdd" Sep 11 20:06:02.113177 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:02.113058 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/be81dfc2-533a-441c-950c-83a3c63a4c6c-webhook-cert\") pod \"opendatahub-operator-controller-manager-5fdf48ff49-xthdd\" (UID: \"be81dfc2-533a-441c-950c-83a3c63a4c6c\") " pod="opendatahub/opendatahub-operator-controller-manager-5fdf48ff49-xthdd" Sep 11 20:06:02.113177 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:02.113078 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5fkw\" (UniqueName: \"kubernetes.io/projected/be81dfc2-533a-441c-950c-83a3c63a4c6c-kube-api-access-m5fkw\") pod \"opendatahub-operator-controller-manager-5fdf48ff49-xthdd\" (UID: \"be81dfc2-533a-441c-950c-83a3c63a4c6c\") " pod="opendatahub/opendatahub-operator-controller-manager-5fdf48ff49-xthdd" Sep 11 20:06:02.113274 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:02.113207 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/be81dfc2-533a-441c-950c-83a3c63a4c6c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5fdf48ff49-xthdd\" (UID: \"be81dfc2-533a-441c-950c-83a3c63a4c6c\") " pod="opendatahub/opendatahub-operator-controller-manager-5fdf48ff49-xthdd" Sep 11 20:06:02.113419 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:02.113395 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/be81dfc2-533a-441c-950c-83a3c63a4c6c-manifests\") pod \"opendatahub-operator-controller-manager-5fdf48ff49-xthdd\" (UID: \"be81dfc2-533a-441c-950c-83a3c63a4c6c\") " pod="opendatahub/opendatahub-operator-controller-manager-5fdf48ff49-xthdd" Sep 11 20:06:02.115395 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:02.115361 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/be81dfc2-533a-441c-950c-83a3c63a4c6c-tmp\") pod \"opendatahub-operator-controller-manager-5fdf48ff49-xthdd\" (UID: \"be81dfc2-533a-441c-950c-83a3c63a4c6c\") " pod="opendatahub/opendatahub-operator-controller-manager-5fdf48ff49-xthdd" Sep 11 20:06:02.115497 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:02.115478 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/be81dfc2-533a-441c-950c-83a3c63a4c6c-webhook-cert\") pod \"opendatahub-operator-controller-manager-5fdf48ff49-xthdd\" (UID: \"be81dfc2-533a-441c-950c-83a3c63a4c6c\") " pod="opendatahub/opendatahub-operator-controller-manager-5fdf48ff49-xthdd" Sep 11 20:06:02.115552 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:02.115493 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/be81dfc2-533a-441c-950c-83a3c63a4c6c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5fdf48ff49-xthdd\" (UID: \"be81dfc2-533a-441c-950c-83a3c63a4c6c\") " pod="opendatahub/opendatahub-operator-controller-manager-5fdf48ff49-xthdd" Sep 11 20:06:02.125789 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:02.125766 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5fkw\" (UniqueName: \"kubernetes.io/projected/be81dfc2-533a-441c-950c-83a3c63a4c6c-kube-api-access-m5fkw\") pod \"opendatahub-operator-controller-manager-5fdf48ff49-xthdd\" (UID: \"be81dfc2-533a-441c-950c-83a3c63a4c6c\") " pod="opendatahub/opendatahub-operator-controller-manager-5fdf48ff49-xthdd" Sep 11 20:06:02.150716 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:02.150691 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:02.250758 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:02.250736 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:02.268787 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:02.268761 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5fdf48ff49-xthdd" Sep 11 20:06:02.350775 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:02.350719 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:02.450488 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:02.450392 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:02.526475 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:02.526447 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:02.544714 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:02.544680 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5fdf48ff49-xthdd"] Sep 11 20:06:02.548462 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:06:02.548436 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe81dfc2_533a_441c_950c_83a3c63a4c6c.slice/crio-b59a8bf86210e3b2a0847b46c24d0c0ef9996aba3ade969dd7593456ac4bae7f WatchSource:0}: Error finding container b59a8bf86210e3b2a0847b46c24d0c0ef9996aba3ade969dd7593456ac4bae7f: Status 404 returned error can't find the container with id b59a8bf86210e3b2a0847b46c24d0c0ef9996aba3ade969dd7593456ac4bae7f Sep 11 20:06:02.597482 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:02.597456 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:02.695567 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:02.695529 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:02.888177 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:02.888148 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5fdf48ff49-xthdd" event={"ID":"be81dfc2-533a-441c-950c-83a3c63a4c6c","Type":"ContainerStarted","Data":"b59a8bf86210e3b2a0847b46c24d0c0ef9996aba3ade969dd7593456ac4bae7f"} Sep 11 20:06:05.903602 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:05.903553 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5fdf48ff49-xthdd" event={"ID":"be81dfc2-533a-441c-950c-83a3c63a4c6c","Type":"ContainerStarted","Data":"a7f6005beb8aabca2c47ca47e7995e5df89e4aedb5573dcc85f23b57c0c97cae"} Sep 11 20:06:06.909675 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:06.909641 2546 generic.go:356] "Generic (PLEG): container finished" podID="be81dfc2-533a-441c-950c-83a3c63a4c6c" containerID="a7f6005beb8aabca2c47ca47e7995e5df89e4aedb5573dcc85f23b57c0c97cae" exitCode=0 Sep 11 20:06:06.910026 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:06.909676 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5fdf48ff49-xthdd" event={"ID":"be81dfc2-533a-441c-950c-83a3c63a4c6c","Type":"ContainerDied","Data":"a7f6005beb8aabca2c47ca47e7995e5df89e4aedb5573dcc85f23b57c0c97cae"} Sep 11 20:06:07.915426 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:07.915396 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5fdf48ff49-xthdd" event={"ID":"be81dfc2-533a-441c-950c-83a3c63a4c6c","Type":"ContainerStarted","Data":"2c1f0075df0e0febc462ec3ac72cdd5743c72f479a2d91baadb61ef8bf479bd8"} Sep 11 20:06:07.915817 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:07.915508 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5fdf48ff49-xthdd" Sep 11 20:06:07.939247 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:07.939202 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5fdf48ff49-xthdd" podStartSLOduration=2.383111287 podStartE2EDuration="6.939185651s" totalImagesPullingTime="3.531345214s" totalInitContainerRuntime="1.02472915s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:06:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:06:07.938732356 +0000 UTC m=+337.986782701" watchObservedRunningTime="2026-09-11 20:06:07.939185651 +0000 UTC m=+337.987235999" Sep 11 20:06:11.800395 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:11.800363 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:11.902261 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:11.902236 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:18.921711 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:18.921675 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5fdf48ff49-xthdd" Sep 11 20:06:21.437702 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:21.437664 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:21.559056 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:21.559018 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:21.850645 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:21.850612 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:22.069147 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:22.069118 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-567ff9dcd5-xpbxn"] Sep 11 20:06:22.077255 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:22.077237 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-567ff9dcd5-xpbxn" Sep 11 20:06:22.079320 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:22.079301 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 11 20:06:22.079868 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:22.079839 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 11 20:06:22.080044 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:22.080030 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 11 20:06:22.080557 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:22.080543 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 11 20:06:22.080614 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:22.080554 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-wjqrv\"" Sep 11 20:06:22.093277 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:22.093258 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-567ff9dcd5-xpbxn"] Sep 11 20:06:22.176696 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:22.176613 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-785lv\" (UniqueName: \"kubernetes.io/projected/3c3d1d29-13cf-48b4-b5a5-4da836cc47cb-kube-api-access-785lv\") pod \"kube-auth-proxy-567ff9dcd5-xpbxn\" (UID: \"3c3d1d29-13cf-48b4-b5a5-4da836cc47cb\") " pod="openshift-ingress/kube-auth-proxy-567ff9dcd5-xpbxn" Sep 11 20:06:22.176696 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:22.176646 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3c3d1d29-13cf-48b4-b5a5-4da836cc47cb-tmp\") pod \"kube-auth-proxy-567ff9dcd5-xpbxn\" (UID: \"3c3d1d29-13cf-48b4-b5a5-4da836cc47cb\") " pod="openshift-ingress/kube-auth-proxy-567ff9dcd5-xpbxn" Sep 11 20:06:22.176696 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:22.176674 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c3d1d29-13cf-48b4-b5a5-4da836cc47cb-tls-certs\") pod \"kube-auth-proxy-567ff9dcd5-xpbxn\" (UID: \"3c3d1d29-13cf-48b4-b5a5-4da836cc47cb\") " pod="openshift-ingress/kube-auth-proxy-567ff9dcd5-xpbxn" Sep 11 20:06:22.277461 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:22.277430 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c3d1d29-13cf-48b4-b5a5-4da836cc47cb-tls-certs\") pod \"kube-auth-proxy-567ff9dcd5-xpbxn\" (UID: \"3c3d1d29-13cf-48b4-b5a5-4da836cc47cb\") " pod="openshift-ingress/kube-auth-proxy-567ff9dcd5-xpbxn" Sep 11 20:06:22.277649 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:22.277519 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-785lv\" (UniqueName: \"kubernetes.io/projected/3c3d1d29-13cf-48b4-b5a5-4da836cc47cb-kube-api-access-785lv\") pod \"kube-auth-proxy-567ff9dcd5-xpbxn\" (UID: \"3c3d1d29-13cf-48b4-b5a5-4da836cc47cb\") " pod="openshift-ingress/kube-auth-proxy-567ff9dcd5-xpbxn" Sep 11 20:06:22.277649 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:22.277546 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3c3d1d29-13cf-48b4-b5a5-4da836cc47cb-tmp\") pod \"kube-auth-proxy-567ff9dcd5-xpbxn\" (UID: \"3c3d1d29-13cf-48b4-b5a5-4da836cc47cb\") " pod="openshift-ingress/kube-auth-proxy-567ff9dcd5-xpbxn" Sep 11 20:06:22.279949 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:22.279923 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3c3d1d29-13cf-48b4-b5a5-4da836cc47cb-tmp\") pod \"kube-auth-proxy-567ff9dcd5-xpbxn\" (UID: \"3c3d1d29-13cf-48b4-b5a5-4da836cc47cb\") " pod="openshift-ingress/kube-auth-proxy-567ff9dcd5-xpbxn" Sep 11 20:06:22.280060 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:22.279981 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c3d1d29-13cf-48b4-b5a5-4da836cc47cb-tls-certs\") pod \"kube-auth-proxy-567ff9dcd5-xpbxn\" (UID: \"3c3d1d29-13cf-48b4-b5a5-4da836cc47cb\") " pod="openshift-ingress/kube-auth-proxy-567ff9dcd5-xpbxn" Sep 11 20:06:22.287431 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:22.287396 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:22.289168 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:22.289146 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-785lv\" (UniqueName: \"kubernetes.io/projected/3c3d1d29-13cf-48b4-b5a5-4da836cc47cb-kube-api-access-785lv\") pod \"kube-auth-proxy-567ff9dcd5-xpbxn\" (UID: \"3c3d1d29-13cf-48b4-b5a5-4da836cc47cb\") " pod="openshift-ingress/kube-auth-proxy-567ff9dcd5-xpbxn" Sep 11 20:06:22.387513 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:22.387489 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-567ff9dcd5-xpbxn" Sep 11 20:06:22.662309 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:22.662282 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-567ff9dcd5-xpbxn"] Sep 11 20:06:22.665728 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:06:22.665701 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c3d1d29_13cf_48b4_b5a5_4da836cc47cb.slice/crio-17a6ab2965d6be464b3184dd67917064d0169a9264660227a39cec80fb488cd1 WatchSource:0}: Error finding container 17a6ab2965d6be464b3184dd67917064d0169a9264660227a39cec80fb488cd1: Status 404 returned error can't find the container with id 17a6ab2965d6be464b3184dd67917064d0169a9264660227a39cec80fb488cd1 Sep 11 20:06:22.980458 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:22.980426 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-567ff9dcd5-xpbxn" event={"ID":"3c3d1d29-13cf-48b4-b5a5-4da836cc47cb","Type":"ContainerStarted","Data":"17a6ab2965d6be464b3184dd67917064d0169a9264660227a39cec80fb488cd1"} Sep 11 20:06:23.428865 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:23.428792 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:23.526253 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:23.526191 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:24.405734 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:24.405699 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:25.998205 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:25.998172 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-567ff9dcd5-xpbxn" event={"ID":"3c3d1d29-13cf-48b4-b5a5-4da836cc47cb","Type":"ContainerStarted","Data":"bc9381c8db2f716ca682072a6eb6f36c02d4a469c633e1450f9367ba3c58c043"} Sep 11 20:06:26.017552 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:26.017508 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-567ff9dcd5-xpbxn" podStartSLOduration=1.118886126 podStartE2EDuration="4.017490746s" totalImagesPullingTime="2.89860462s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:06:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:06:26.016494896 +0000 UTC m=+356.064545250" watchObservedRunningTime="2026-09-11 20:06:26.017490746 +0000 UTC m=+356.065541091" Sep 11 20:06:28.499083 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:28.499049 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:28.600477 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:28.600445 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:28.757221 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:28.757153 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk"] Sep 11 20:06:28.761067 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:28.761047 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk" Sep 11 20:06:28.763136 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:28.763089 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-7fvxf\"" Sep 11 20:06:28.763224 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:28.763207 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 11 20:06:28.771793 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:28.771772 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk"] Sep 11 20:06:28.803981 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:28.803957 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:28.835288 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:28.835260 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b98b581-3bb6-4c00-8da1-01e45c20c236-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk\" (UID: \"8b98b581-3bb6-4c00-8da1-01e45c20c236\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk" Sep 11 20:06:28.835390 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:28.835305 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b98b581-3bb6-4c00-8da1-01e45c20c236-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk\" (UID: \"8b98b581-3bb6-4c00-8da1-01e45c20c236\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk" Sep 11 20:06:28.835390 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:28.835337 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b98b581-3bb6-4c00-8da1-01e45c20c236-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk\" (UID: \"8b98b581-3bb6-4c00-8da1-01e45c20c236\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk" Sep 11 20:06:28.835510 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:28.835397 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b98b581-3bb6-4c00-8da1-01e45c20c236-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk\" (UID: \"8b98b581-3bb6-4c00-8da1-01e45c20c236\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk" Sep 11 20:06:28.835510 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:28.835447 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b98b581-3bb6-4c00-8da1-01e45c20c236-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk\" (UID: \"8b98b581-3bb6-4c00-8da1-01e45c20c236\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk" Sep 11 20:06:28.835510 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:28.835478 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b98b581-3bb6-4c00-8da1-01e45c20c236-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk\" (UID: \"8b98b581-3bb6-4c00-8da1-01e45c20c236\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk" Sep 11 20:06:28.835661 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:28.835506 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dt88\" (UniqueName: \"kubernetes.io/projected/8b98b581-3bb6-4c00-8da1-01e45c20c236-kube-api-access-5dt88\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk\" (UID: \"8b98b581-3bb6-4c00-8da1-01e45c20c236\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk" Sep 11 20:06:28.835661 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:28.835567 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b98b581-3bb6-4c00-8da1-01e45c20c236-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk\" (UID: \"8b98b581-3bb6-4c00-8da1-01e45c20c236\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk" Sep 11 20:06:28.835661 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:28.835623 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b98b581-3bb6-4c00-8da1-01e45c20c236-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk\" (UID: \"8b98b581-3bb6-4c00-8da1-01e45c20c236\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk" Sep 11 20:06:28.936759 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:28.936729 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b98b581-3bb6-4c00-8da1-01e45c20c236-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk\" (UID: \"8b98b581-3bb6-4c00-8da1-01e45c20c236\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk" Sep 11 20:06:28.936759 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:28.936764 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b98b581-3bb6-4c00-8da1-01e45c20c236-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk\" (UID: \"8b98b581-3bb6-4c00-8da1-01e45c20c236\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk" Sep 11 20:06:28.936985 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:28.936788 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b98b581-3bb6-4c00-8da1-01e45c20c236-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk\" (UID: \"8b98b581-3bb6-4c00-8da1-01e45c20c236\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk" Sep 11 20:06:28.936985 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:28.936809 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b98b581-3bb6-4c00-8da1-01e45c20c236-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk\" (UID: \"8b98b581-3bb6-4c00-8da1-01e45c20c236\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk" Sep 11 20:06:28.936985 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:28.936850 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b98b581-3bb6-4c00-8da1-01e45c20c236-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk\" (UID: \"8b98b581-3bb6-4c00-8da1-01e45c20c236\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk" Sep 11 20:06:28.936985 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:28.936922 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b98b581-3bb6-4c00-8da1-01e45c20c236-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk\" (UID: \"8b98b581-3bb6-4c00-8da1-01e45c20c236\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk" Sep 11 20:06:28.936985 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:28.936958 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b98b581-3bb6-4c00-8da1-01e45c20c236-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk\" (UID: \"8b98b581-3bb6-4c00-8da1-01e45c20c236\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk" Sep 11 20:06:28.937253 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:28.936990 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dt88\" (UniqueName: \"kubernetes.io/projected/8b98b581-3bb6-4c00-8da1-01e45c20c236-kube-api-access-5dt88\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk\" (UID: \"8b98b581-3bb6-4c00-8da1-01e45c20c236\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk" Sep 11 20:06:28.937253 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:28.937048 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b98b581-3bb6-4c00-8da1-01e45c20c236-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk\" (UID: \"8b98b581-3bb6-4c00-8da1-01e45c20c236\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk" Sep 11 20:06:28.937253 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:28.937213 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b98b581-3bb6-4c00-8da1-01e45c20c236-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk\" (UID: \"8b98b581-3bb6-4c00-8da1-01e45c20c236\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk" Sep 11 20:06:28.937413 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:28.937295 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b98b581-3bb6-4c00-8da1-01e45c20c236-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk\" (UID: \"8b98b581-3bb6-4c00-8da1-01e45c20c236\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk" Sep 11 20:06:28.937413 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:28.937379 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b98b581-3bb6-4c00-8da1-01e45c20c236-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk\" (UID: \"8b98b581-3bb6-4c00-8da1-01e45c20c236\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk" Sep 11 20:06:28.937561 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:28.937539 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b98b581-3bb6-4c00-8da1-01e45c20c236-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk\" (UID: \"8b98b581-3bb6-4c00-8da1-01e45c20c236\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk" Sep 11 20:06:28.937645 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:28.937630 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b98b581-3bb6-4c00-8da1-01e45c20c236-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk\" (UID: \"8b98b581-3bb6-4c00-8da1-01e45c20c236\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk" Sep 11 20:06:28.939631 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:28.939606 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b98b581-3bb6-4c00-8da1-01e45c20c236-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk\" (UID: \"8b98b581-3bb6-4c00-8da1-01e45c20c236\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk" Sep 11 20:06:28.939747 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:28.939730 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b98b581-3bb6-4c00-8da1-01e45c20c236-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk\" (UID: \"8b98b581-3bb6-4c00-8da1-01e45c20c236\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk" Sep 11 20:06:28.948818 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:28.948793 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b98b581-3bb6-4c00-8da1-01e45c20c236-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk\" (UID: \"8b98b581-3bb6-4c00-8da1-01e45c20c236\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk" Sep 11 20:06:28.949052 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:28.949029 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dt88\" (UniqueName: \"kubernetes.io/projected/8b98b581-3bb6-4c00-8da1-01e45c20c236-kube-api-access-5dt88\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk\" (UID: \"8b98b581-3bb6-4c00-8da1-01e45c20c236\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk" Sep 11 20:06:29.011458 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:29.011410 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:29.070458 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:29.070436 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk" Sep 11 20:06:29.195905 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:29.195872 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:29.346441 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:29.346408 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk"] Sep 11 20:06:29.349265 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:06:29.349239 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b98b581_3bb6_4c00_8da1_01e45c20c236.slice/crio-61c14c3d12fe30e9ae4ad9d7b40a22dd201813e14b63375103fed52d2a2997ea WatchSource:0}: Error finding container 61c14c3d12fe30e9ae4ad9d7b40a22dd201813e14b63375103fed52d2a2997ea: Status 404 returned error can't find the container with id 61c14c3d12fe30e9ae4ad9d7b40a22dd201813e14b63375103fed52d2a2997ea Sep 11 20:06:30.015663 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:30.015631 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk" event={"ID":"8b98b581-3bb6-4c00-8da1-01e45c20c236","Type":"ContainerStarted","Data":"61c14c3d12fe30e9ae4ad9d7b40a22dd201813e14b63375103fed52d2a2997ea"} Sep 11 20:06:32.577423 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:32.577382 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-htbhw"] Sep 11 20:06:32.585004 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:32.584974 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-htbhw" Sep 11 20:06:32.587042 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:32.587015 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Sep 11 20:06:32.587167 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:32.587046 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-2qtsn\"" Sep 11 20:06:32.591358 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:32.591316 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-htbhw"] Sep 11 20:06:32.607753 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:32.607727 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:32.668138 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:32.668096 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0c85ece7-fbaf-4a92-99d5-e48a0170cd2d-cert\") pod \"odh-model-controller-7998c67dc4-htbhw\" (UID: \"0c85ece7-fbaf-4a92-99d5-e48a0170cd2d\") " pod="opendatahub/odh-model-controller-7998c67dc4-htbhw" Sep 11 20:06:32.668297 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:32.668191 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48vxs\" (UniqueName: \"kubernetes.io/projected/0c85ece7-fbaf-4a92-99d5-e48a0170cd2d-kube-api-access-48vxs\") pod \"odh-model-controller-7998c67dc4-htbhw\" (UID: \"0c85ece7-fbaf-4a92-99d5-e48a0170cd2d\") " pod="opendatahub/odh-model-controller-7998c67dc4-htbhw" Sep 11 20:06:32.676277 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:32.676241 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:32.768751 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:32.768715 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48vxs\" (UniqueName: \"kubernetes.io/projected/0c85ece7-fbaf-4a92-99d5-e48a0170cd2d-kube-api-access-48vxs\") pod \"odh-model-controller-7998c67dc4-htbhw\" (UID: \"0c85ece7-fbaf-4a92-99d5-e48a0170cd2d\") " pod="opendatahub/odh-model-controller-7998c67dc4-htbhw" Sep 11 20:06:32.768936 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:32.768812 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0c85ece7-fbaf-4a92-99d5-e48a0170cd2d-cert\") pod \"odh-model-controller-7998c67dc4-htbhw\" (UID: \"0c85ece7-fbaf-4a92-99d5-e48a0170cd2d\") " pod="opendatahub/odh-model-controller-7998c67dc4-htbhw" Sep 11 20:06:32.768936 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:06:32.768922 2546 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Sep 11 20:06:32.769054 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:06:32.768983 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c85ece7-fbaf-4a92-99d5-e48a0170cd2d-cert podName:0c85ece7-fbaf-4a92-99d5-e48a0170cd2d nodeName:}" failed. No retries permitted until 2026-09-11 20:06:33.268964128 +0000 UTC m=+363.317014461 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/0c85ece7-fbaf-4a92-99d5-e48a0170cd2d-cert") pod "odh-model-controller-7998c67dc4-htbhw" (UID: "0c85ece7-fbaf-4a92-99d5-e48a0170cd2d") : secret "odh-model-controller-webhook-cert" not found Sep 11 20:06:32.775866 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:32.775836 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:32.777625 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:32.777595 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48vxs\" (UniqueName: \"kubernetes.io/projected/0c85ece7-fbaf-4a92-99d5-e48a0170cd2d-kube-api-access-48vxs\") pod \"odh-model-controller-7998c67dc4-htbhw\" (UID: \"0c85ece7-fbaf-4a92-99d5-e48a0170cd2d\") " pod="opendatahub/odh-model-controller-7998c67dc4-htbhw" Sep 11 20:06:32.861639 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:32.861555 2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 20:06:32.861774 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:32.861674 2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 20:06:32.861774 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:32.861704 2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 20:06:32.965266 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:32.965228 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:33.033355 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:33.033317 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk" event={"ID":"8b98b581-3bb6-4c00-8da1-01e45c20c236","Type":"ContainerStarted","Data":"8a5701c66c4d52f7c15df0a5bf68bb4232e3469f5e845858097bd0905c4deee5"} Sep 11 20:06:33.053917 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:33.053858 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk" podStartSLOduration=1.543700833 podStartE2EDuration="5.053838477s" totalImagesPullingTime="3.510137644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:06:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:06:33.053198499 +0000 UTC m=+363.101248841" watchObservedRunningTime="2026-09-11 20:06:33.053838477 +0000 UTC m=+363.101888822" Sep 11 20:06:33.071515 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:33.071486 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk" Sep 11 20:06:33.073166 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:33.073133 2546 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body= Sep 11 20:06:33.073311 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:33.073194 2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk" podUID="8b98b581-3bb6-4c00-8da1-01e45c20c236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" Sep 11 20:06:33.205283 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:33.205196 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:33.217968 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:33.217947 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-hmfh4"] Sep 11 20:06:33.221223 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:33.221209 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-hmfh4" Sep 11 20:06:33.223113 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:33.223095 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Sep 11 20:06:33.223198 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:33.223100 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-h26dl\"" Sep 11 20:06:33.228592 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:33.228557 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-hmfh4"] Sep 11 20:06:33.274235 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:33.274204 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0c85ece7-fbaf-4a92-99d5-e48a0170cd2d-cert\") pod \"odh-model-controller-7998c67dc4-htbhw\" (UID: \"0c85ece7-fbaf-4a92-99d5-e48a0170cd2d\") " pod="opendatahub/odh-model-controller-7998c67dc4-htbhw" Sep 11 20:06:33.276687 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:33.276663 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0c85ece7-fbaf-4a92-99d5-e48a0170cd2d-cert\") pod \"odh-model-controller-7998c67dc4-htbhw\" (UID: \"0c85ece7-fbaf-4a92-99d5-e48a0170cd2d\") " pod="opendatahub/odh-model-controller-7998c67dc4-htbhw" Sep 11 20:06:33.305733 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:33.305700 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:33.375027 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:33.374996 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7fkg\" (UniqueName: \"kubernetes.io/projected/30b6578d-630f-4290-ad87-b335c2eef845-kube-api-access-s7fkg\") pod \"model-serving-api-54d89bd5fc-hmfh4\" (UID: \"30b6578d-630f-4290-ad87-b335c2eef845\") " pod="opendatahub/model-serving-api-54d89bd5fc-hmfh4" Sep 11 20:06:33.375162 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:33.375033 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30b6578d-630f-4290-ad87-b335c2eef845-tls-certs\") pod \"model-serving-api-54d89bd5fc-hmfh4\" (UID: \"30b6578d-630f-4290-ad87-b335c2eef845\") " pod="opendatahub/model-serving-api-54d89bd5fc-hmfh4" Sep 11 20:06:33.476023 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:33.475936 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7fkg\" (UniqueName: \"kubernetes.io/projected/30b6578d-630f-4290-ad87-b335c2eef845-kube-api-access-s7fkg\") pod \"model-serving-api-54d89bd5fc-hmfh4\" (UID: \"30b6578d-630f-4290-ad87-b335c2eef845\") " pod="opendatahub/model-serving-api-54d89bd5fc-hmfh4" Sep 11 20:06:33.476023 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:33.475987 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30b6578d-630f-4290-ad87-b335c2eef845-tls-certs\") pod \"model-serving-api-54d89bd5fc-hmfh4\" (UID: \"30b6578d-630f-4290-ad87-b335c2eef845\") " pod="opendatahub/model-serving-api-54d89bd5fc-hmfh4" Sep 11 20:06:33.478611 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:33.478557 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30b6578d-630f-4290-ad87-b335c2eef845-tls-certs\") pod \"model-serving-api-54d89bd5fc-hmfh4\" (UID: \"30b6578d-630f-4290-ad87-b335c2eef845\") " pod="opendatahub/model-serving-api-54d89bd5fc-hmfh4" Sep 11 20:06:33.485781 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:33.485756 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7fkg\" (UniqueName: \"kubernetes.io/projected/30b6578d-630f-4290-ad87-b335c2eef845-kube-api-access-s7fkg\") pod \"model-serving-api-54d89bd5fc-hmfh4\" (UID: \"30b6578d-630f-4290-ad87-b335c2eef845\") " pod="opendatahub/model-serving-api-54d89bd5fc-hmfh4" Sep 11 20:06:33.499625 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:33.499599 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-htbhw" Sep 11 20:06:33.531728 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:33.531696 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-hmfh4" Sep 11 20:06:33.776247 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:33.776222 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-htbhw"] Sep 11 20:06:33.779083 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:06:33.779049 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c85ece7_fbaf_4a92_99d5_e48a0170cd2d.slice/crio-6b83846b00ec1768d18b5102cf8e6e88a585a09e86efa09b34f5f2b57b26d21c WatchSource:0}: Error finding container 6b83846b00ec1768d18b5102cf8e6e88a585a09e86efa09b34f5f2b57b26d21c: Status 404 returned error can't find the container with id 6b83846b00ec1768d18b5102cf8e6e88a585a09e86efa09b34f5f2b57b26d21c Sep 11 20:06:33.815166 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:33.815111 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-hmfh4"] Sep 11 20:06:33.817014 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:06:33.816992 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30b6578d_630f_4290_ad87_b335c2eef845.slice/crio-1926839170f510db475e4070983fcd0aa894095989bbaad6777ff9af36ccab0c WatchSource:0}: Error finding container 1926839170f510db475e4070983fcd0aa894095989bbaad6777ff9af36ccab0c: Status 404 returned error can't find the container with id 1926839170f510db475e4070983fcd0aa894095989bbaad6777ff9af36ccab0c Sep 11 20:06:34.040277 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:34.040188 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-hmfh4" event={"ID":"30b6578d-630f-4290-ad87-b335c2eef845","Type":"ContainerStarted","Data":"1926839170f510db475e4070983fcd0aa894095989bbaad6777ff9af36ccab0c"} Sep 11 20:06:34.041499 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:34.041463 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-htbhw" event={"ID":"0c85ece7-fbaf-4a92-99d5-e48a0170cd2d","Type":"ContainerStarted","Data":"6b83846b00ec1768d18b5102cf8e6e88a585a09e86efa09b34f5f2b57b26d21c"} Sep 11 20:06:34.071114 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:34.071085 2546 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body= Sep 11 20:06:34.071254 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:34.071142 2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk" podUID="8b98b581-3bb6-4c00-8da1-01e45c20c236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" Sep 11 20:06:34.126380 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:34.126328 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk"] Sep 11 20:06:35.071079 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:35.071036 2546 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body= Sep 11 20:06:35.071545 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:35.071107 2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk" podUID="8b98b581-3bb6-4c00-8da1-01e45c20c236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" Sep 11 20:06:36.059973 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:36.059898 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk" podUID="8b98b581-3bb6-4c00-8da1-01e45c20c236" containerName="istio-proxy" containerID="cri-o://8a5701c66c4d52f7c15df0a5bf68bb4232e3469f5e845858097bd0905c4deee5" gracePeriod=30 Sep 11 20:06:38.072083 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:38.072045 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-htbhw" event={"ID":"0c85ece7-fbaf-4a92-99d5-e48a0170cd2d","Type":"ContainerStarted","Data":"ddc3eec18fa53faef91ef7cc0c7d6ead9721b18fc1c74e3ee07afa2b30faaff6"} Sep 11 20:06:38.072514 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:38.072150 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-htbhw" Sep 11 20:06:38.073635 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:38.073610 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-hmfh4" event={"ID":"30b6578d-630f-4290-ad87-b335c2eef845","Type":"ContainerStarted","Data":"1ab29ef7fd606485bbd070371bf2fc6a29785068e6119f341b921a654f2bd59d"} Sep 11 20:06:38.073764 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:38.073671 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-hmfh4" Sep 11 20:06:38.087535 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:38.087493 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-htbhw" podStartSLOduration=2.528066903 podStartE2EDuration="6.087479983s" totalImagesPullingTime="3.55941308s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:06:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:06:38.086590166 +0000 UTC m=+368.134640502" watchObservedRunningTime="2026-09-11 20:06:38.087479983 +0000 UTC m=+368.135530327" Sep 11 20:06:38.102612 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:38.102551 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-hmfh4" podStartSLOduration=1.5771513480000001 podStartE2EDuration="5.102539754s" totalImagesPullingTime="3.525388406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:06:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:06:38.10185125 +0000 UTC m=+368.149901596" watchObservedRunningTime="2026-09-11 20:06:38.102539754 +0000 UTC m=+368.150590124" Sep 11 20:06:38.230301 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:38.230271 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7c4d7c7587-68mg4"] Sep 11 20:06:38.232720 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:38.232704 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c4d7c7587-68mg4" Sep 11 20:06:38.234893 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:38.234870 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 11 20:06:38.235087 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:38.234960 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 11 20:06:38.235202 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:38.235171 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 11 20:06:38.235375 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:38.235220 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 11 20:06:38.235375 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:38.235277 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 11 20:06:38.235375 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:38.235292 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 11 20:06:38.235763 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:38.235701 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-g7fth\"" Sep 11 20:06:38.235763 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:38.235739 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 11 20:06:38.240277 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:38.240255 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 11 20:06:38.245189 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:38.245168 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c4d7c7587-68mg4"] Sep 11 20:06:38.286865 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:38.286789 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:38.319063 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:38.319036 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e5e14f90-ad2f-43f6-92c3-e05d52e97dab-console-serving-cert\") pod \"console-7c4d7c7587-68mg4\" (UID: \"e5e14f90-ad2f-43f6-92c3-e05d52e97dab\") " pod="openshift-console/console-7c4d7c7587-68mg4" Sep 11 20:06:38.319151 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:38.319074 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e5e14f90-ad2f-43f6-92c3-e05d52e97dab-console-config\") pod \"console-7c4d7c7587-68mg4\" (UID: \"e5e14f90-ad2f-43f6-92c3-e05d52e97dab\") " pod="openshift-console/console-7c4d7c7587-68mg4" Sep 11 20:06:38.319151 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:38.319090 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e5e14f90-ad2f-43f6-92c3-e05d52e97dab-service-ca\") pod \"console-7c4d7c7587-68mg4\" (UID: \"e5e14f90-ad2f-43f6-92c3-e05d52e97dab\") " pod="openshift-console/console-7c4d7c7587-68mg4" Sep 11 20:06:38.319151 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:38.319115 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e5e14f90-ad2f-43f6-92c3-e05d52e97dab-oauth-serving-cert\") pod \"console-7c4d7c7587-68mg4\" (UID: \"e5e14f90-ad2f-43f6-92c3-e05d52e97dab\") " pod="openshift-console/console-7c4d7c7587-68mg4" Sep 11 20:06:38.319271 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:38.319187 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5e14f90-ad2f-43f6-92c3-e05d52e97dab-trusted-ca-bundle\") pod \"console-7c4d7c7587-68mg4\" (UID: \"e5e14f90-ad2f-43f6-92c3-e05d52e97dab\") " pod="openshift-console/console-7c4d7c7587-68mg4" Sep 11 20:06:38.319271 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:38.319254 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e5e14f90-ad2f-43f6-92c3-e05d52e97dab-console-oauth-config\") pod \"console-7c4d7c7587-68mg4\" (UID: \"e5e14f90-ad2f-43f6-92c3-e05d52e97dab\") " pod="openshift-console/console-7c4d7c7587-68mg4" Sep 11 20:06:38.319340 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:38.319294 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dsnl\" (UniqueName: \"kubernetes.io/projected/e5e14f90-ad2f-43f6-92c3-e05d52e97dab-kube-api-access-4dsnl\") pod \"console-7c4d7c7587-68mg4\" (UID: \"e5e14f90-ad2f-43f6-92c3-e05d52e97dab\") " pod="openshift-console/console-7c4d7c7587-68mg4" Sep 11 20:06:38.344832 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:38.344781 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:38.419936 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:38.419901 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e5e14f90-ad2f-43f6-92c3-e05d52e97dab-console-config\") pod \"console-7c4d7c7587-68mg4\" (UID: \"e5e14f90-ad2f-43f6-92c3-e05d52e97dab\") " pod="openshift-console/console-7c4d7c7587-68mg4" Sep 11 20:06:38.419936 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:38.419933 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e5e14f90-ad2f-43f6-92c3-e05d52e97dab-service-ca\") pod \"console-7c4d7c7587-68mg4\" (UID: \"e5e14f90-ad2f-43f6-92c3-e05d52e97dab\") " pod="openshift-console/console-7c4d7c7587-68mg4" Sep 11 20:06:38.420152 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:38.419958 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e5e14f90-ad2f-43f6-92c3-e05d52e97dab-oauth-serving-cert\") pod \"console-7c4d7c7587-68mg4\" (UID: \"e5e14f90-ad2f-43f6-92c3-e05d52e97dab\") " pod="openshift-console/console-7c4d7c7587-68mg4" Sep 11 20:06:38.420152 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:38.419989 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5e14f90-ad2f-43f6-92c3-e05d52e97dab-trusted-ca-bundle\") pod \"console-7c4d7c7587-68mg4\" (UID: \"e5e14f90-ad2f-43f6-92c3-e05d52e97dab\") " pod="openshift-console/console-7c4d7c7587-68mg4" Sep 11 20:06:38.420152 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:38.420031 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e5e14f90-ad2f-43f6-92c3-e05d52e97dab-console-oauth-config\") pod \"console-7c4d7c7587-68mg4\" (UID: \"e5e14f90-ad2f-43f6-92c3-e05d52e97dab\") " pod="openshift-console/console-7c4d7c7587-68mg4" Sep 11 20:06:38.420152 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:38.420046 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dsnl\" (UniqueName: \"kubernetes.io/projected/e5e14f90-ad2f-43f6-92c3-e05d52e97dab-kube-api-access-4dsnl\") pod \"console-7c4d7c7587-68mg4\" (UID: \"e5e14f90-ad2f-43f6-92c3-e05d52e97dab\") " pod="openshift-console/console-7c4d7c7587-68mg4" Sep 11 20:06:38.420152 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:38.420073 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e5e14f90-ad2f-43f6-92c3-e05d52e97dab-console-serving-cert\") pod \"console-7c4d7c7587-68mg4\" (UID: \"e5e14f90-ad2f-43f6-92c3-e05d52e97dab\") " pod="openshift-console/console-7c4d7c7587-68mg4" Sep 11 20:06:38.420782 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:38.420742 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e5e14f90-ad2f-43f6-92c3-e05d52e97dab-service-ca\") pod \"console-7c4d7c7587-68mg4\" (UID: \"e5e14f90-ad2f-43f6-92c3-e05d52e97dab\") " pod="openshift-console/console-7c4d7c7587-68mg4" Sep 11 20:06:38.420937 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:38.420911 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e5e14f90-ad2f-43f6-92c3-e05d52e97dab-console-config\") pod \"console-7c4d7c7587-68mg4\" (UID: \"e5e14f90-ad2f-43f6-92c3-e05d52e97dab\") " pod="openshift-console/console-7c4d7c7587-68mg4" Sep 11 20:06:38.420937 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:38.420932 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e5e14f90-ad2f-43f6-92c3-e05d52e97dab-oauth-serving-cert\") pod \"console-7c4d7c7587-68mg4\" (UID: \"e5e14f90-ad2f-43f6-92c3-e05d52e97dab\") " pod="openshift-console/console-7c4d7c7587-68mg4" Sep 11 20:06:38.421178 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:38.421159 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5e14f90-ad2f-43f6-92c3-e05d52e97dab-trusted-ca-bundle\") pod \"console-7c4d7c7587-68mg4\" (UID: \"e5e14f90-ad2f-43f6-92c3-e05d52e97dab\") " pod="openshift-console/console-7c4d7c7587-68mg4" Sep 11 20:06:38.422529 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:38.422510 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e5e14f90-ad2f-43f6-92c3-e05d52e97dab-console-serving-cert\") pod \"console-7c4d7c7587-68mg4\" (UID: \"e5e14f90-ad2f-43f6-92c3-e05d52e97dab\") " pod="openshift-console/console-7c4d7c7587-68mg4" Sep 11 20:06:38.422625 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:38.422516 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e5e14f90-ad2f-43f6-92c3-e05d52e97dab-console-oauth-config\") pod \"console-7c4d7c7587-68mg4\" (UID: \"e5e14f90-ad2f-43f6-92c3-e05d52e97dab\") " pod="openshift-console/console-7c4d7c7587-68mg4" Sep 11 20:06:38.427069 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:38.427046 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dsnl\" (UniqueName: \"kubernetes.io/projected/e5e14f90-ad2f-43f6-92c3-e05d52e97dab-kube-api-access-4dsnl\") pod \"console-7c4d7c7587-68mg4\" (UID: \"e5e14f90-ad2f-43f6-92c3-e05d52e97dab\") " pod="openshift-console/console-7c4d7c7587-68mg4" Sep 11 20:06:38.542452 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:38.542414 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c4d7c7587-68mg4" Sep 11 20:06:38.812786 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:38.812764 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c4d7c7587-68mg4"] Sep 11 20:06:38.815212 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:06:38.815186 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5e14f90_ad2f_43f6_92c3_e05d52e97dab.slice/crio-1615abfd1bb0cd2a3550cd20f8d32031202fb5cdc27b9dc1828578372070e515 WatchSource:0}: Error finding container 1615abfd1bb0cd2a3550cd20f8d32031202fb5cdc27b9dc1828578372070e515: Status 404 returned error can't find the container with id 1615abfd1bb0cd2a3550cd20f8d32031202fb5cdc27b9dc1828578372070e515 Sep 11 20:06:39.079342 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:39.079260 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c4d7c7587-68mg4" event={"ID":"e5e14f90-ad2f-43f6-92c3-e05d52e97dab","Type":"ContainerStarted","Data":"7fb8c09dc843b553f88803077d2cefc1dfbe1971d1fe9260530908f31ca004b7"} Sep 11 20:06:39.079750 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:39.079342 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c4d7c7587-68mg4" event={"ID":"e5e14f90-ad2f-43f6-92c3-e05d52e97dab","Type":"ContainerStarted","Data":"1615abfd1bb0cd2a3550cd20f8d32031202fb5cdc27b9dc1828578372070e515"} Sep 11 20:06:39.103600 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:39.103539 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7c4d7c7587-68mg4" podStartSLOduration=1.103519584 podStartE2EDuration="1.103519584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:06:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:06:39.103040802 +0000 UTC m=+369.151091146" watchObservedRunningTime="2026-09-11 20:06:39.103519584 +0000 UTC m=+369.151569931" Sep 11 20:06:39.817211 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:39.817179 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-pnz6j"] Sep 11 20:06:39.819989 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:39.819970 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pnz6j" Sep 11 20:06:39.822096 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:39.822076 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 11 20:06:39.822204 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:39.822185 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 11 20:06:39.822347 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:39.822331 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-gr65x\"" Sep 11 20:06:39.822402 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:39.822390 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 11 20:06:39.834338 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:39.834318 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-pnz6j"] Sep 11 20:06:39.934414 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:39.934372 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d6aef9ef-d852-445e-9538-7e487cdbbf6f-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-pnz6j\" (UID: \"d6aef9ef-d852-445e-9538-7e487cdbbf6f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pnz6j" Sep 11 20:06:39.934602 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:39.934417 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6aef9ef-d852-445e-9538-7e487cdbbf6f-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-pnz6j\" (UID: \"d6aef9ef-d852-445e-9538-7e487cdbbf6f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pnz6j" Sep 11 20:06:39.934602 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:39.934451 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d6aef9ef-d852-445e-9538-7e487cdbbf6f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-pnz6j\" (UID: \"d6aef9ef-d852-445e-9538-7e487cdbbf6f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pnz6j" Sep 11 20:06:39.934602 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:39.934485 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6aef9ef-d852-445e-9538-7e487cdbbf6f-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-pnz6j\" (UID: \"d6aef9ef-d852-445e-9538-7e487cdbbf6f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pnz6j" Sep 11 20:06:39.934602 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:39.934517 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzxdh\" (UniqueName: \"kubernetes.io/projected/d6aef9ef-d852-445e-9538-7e487cdbbf6f-kube-api-access-kzxdh\") pod \"istiod-openshift-gateway-5677ccb49b-pnz6j\" (UID: \"d6aef9ef-d852-445e-9538-7e487cdbbf6f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pnz6j" Sep 11 20:06:39.934800 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:39.934638 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d6aef9ef-d852-445e-9538-7e487cdbbf6f-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-pnz6j\" (UID: \"d6aef9ef-d852-445e-9538-7e487cdbbf6f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pnz6j" Sep 11 20:06:39.934800 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:39.934671 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d6aef9ef-d852-445e-9538-7e487cdbbf6f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-pnz6j\" (UID: \"d6aef9ef-d852-445e-9538-7e487cdbbf6f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pnz6j" Sep 11 20:06:40.035346 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:40.035306 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d6aef9ef-d852-445e-9538-7e487cdbbf6f-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-pnz6j\" (UID: \"d6aef9ef-d852-445e-9538-7e487cdbbf6f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pnz6j" Sep 11 20:06:40.035346 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:40.035350 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d6aef9ef-d852-445e-9538-7e487cdbbf6f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-pnz6j\" (UID: \"d6aef9ef-d852-445e-9538-7e487cdbbf6f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pnz6j" Sep 11 20:06:40.035548 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:40.035452 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d6aef9ef-d852-445e-9538-7e487cdbbf6f-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-pnz6j\" (UID: \"d6aef9ef-d852-445e-9538-7e487cdbbf6f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pnz6j" Sep 11 20:06:40.035548 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:40.035482 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6aef9ef-d852-445e-9538-7e487cdbbf6f-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-pnz6j\" (UID: \"d6aef9ef-d852-445e-9538-7e487cdbbf6f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pnz6j" Sep 11 20:06:40.035548 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:40.035504 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d6aef9ef-d852-445e-9538-7e487cdbbf6f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-pnz6j\" (UID: \"d6aef9ef-d852-445e-9538-7e487cdbbf6f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pnz6j" Sep 11 20:06:40.035548 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:40.035531 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6aef9ef-d852-445e-9538-7e487cdbbf6f-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-pnz6j\" (UID: \"d6aef9ef-d852-445e-9538-7e487cdbbf6f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pnz6j" Sep 11 20:06:40.035824 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:40.035565 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzxdh\" (UniqueName: \"kubernetes.io/projected/d6aef9ef-d852-445e-9538-7e487cdbbf6f-kube-api-access-kzxdh\") pod \"istiod-openshift-gateway-5677ccb49b-pnz6j\" (UID: \"d6aef9ef-d852-445e-9538-7e487cdbbf6f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pnz6j" Sep 11 20:06:40.036349 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:40.036322 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d6aef9ef-d852-445e-9538-7e487cdbbf6f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-pnz6j\" (UID: \"d6aef9ef-d852-445e-9538-7e487cdbbf6f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pnz6j" Sep 11 20:06:40.037768 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:40.037740 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6aef9ef-d852-445e-9538-7e487cdbbf6f-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-pnz6j\" (UID: \"d6aef9ef-d852-445e-9538-7e487cdbbf6f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pnz6j" Sep 11 20:06:40.037850 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:40.037819 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d6aef9ef-d852-445e-9538-7e487cdbbf6f-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-pnz6j\" (UID: \"d6aef9ef-d852-445e-9538-7e487cdbbf6f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pnz6j" Sep 11 20:06:40.037886 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:40.037852 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d6aef9ef-d852-445e-9538-7e487cdbbf6f-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-pnz6j\" (UID: \"d6aef9ef-d852-445e-9538-7e487cdbbf6f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pnz6j" Sep 11 20:06:40.037945 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:40.037930 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d6aef9ef-d852-445e-9538-7e487cdbbf6f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-pnz6j\" (UID: \"d6aef9ef-d852-445e-9538-7e487cdbbf6f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pnz6j" Sep 11 20:06:40.049832 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:40.049813 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6aef9ef-d852-445e-9538-7e487cdbbf6f-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-pnz6j\" (UID: \"d6aef9ef-d852-445e-9538-7e487cdbbf6f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pnz6j" Sep 11 20:06:40.057031 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:40.057007 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzxdh\" (UniqueName: \"kubernetes.io/projected/d6aef9ef-d852-445e-9538-7e487cdbbf6f-kube-api-access-kzxdh\") pod \"istiod-openshift-gateway-5677ccb49b-pnz6j\" (UID: \"d6aef9ef-d852-445e-9538-7e487cdbbf6f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pnz6j" Sep 11 20:06:40.129931 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:40.129905 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pnz6j" Sep 11 20:06:40.428384 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:40.428282 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-pnz6j"] Sep 11 20:06:40.430540 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:06:40.430507 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6aef9ef_d852_445e_9538_7e487cdbbf6f.slice/crio-1bde381da917c12b38a3fe250045b0e803ac89b3ebde1759b962481f563c4117 WatchSource:0}: Error finding container 1bde381da917c12b38a3fe250045b0e803ac89b3ebde1759b962481f563c4117: Status 404 returned error can't find the container with id 1bde381da917c12b38a3fe250045b0e803ac89b3ebde1759b962481f563c4117 Sep 11 20:06:41.088972 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:41.088945 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pnz6j" event={"ID":"d6aef9ef-d852-445e-9538-7e487cdbbf6f","Type":"ContainerStarted","Data":"1bde381da917c12b38a3fe250045b0e803ac89b3ebde1759b962481f563c4117"} Sep 11 20:06:41.311608 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:41.311566 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk" Sep 11 20:06:41.449535 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:41.449497 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b98b581-3bb6-4c00-8da1-01e45c20c236-istio-podinfo\") pod \"8b98b581-3bb6-4c00-8da1-01e45c20c236\" (UID: \"8b98b581-3bb6-4c00-8da1-01e45c20c236\") " Sep 11 20:06:41.449750 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:41.449589 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b98b581-3bb6-4c00-8da1-01e45c20c236-istiod-ca-cert\") pod \"8b98b581-3bb6-4c00-8da1-01e45c20c236\" (UID: \"8b98b581-3bb6-4c00-8da1-01e45c20c236\") " Sep 11 20:06:41.449750 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:41.449621 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5dt88\" (UniqueName: \"kubernetes.io/projected/8b98b581-3bb6-4c00-8da1-01e45c20c236-kube-api-access-5dt88\") pod \"8b98b581-3bb6-4c00-8da1-01e45c20c236\" (UID: \"8b98b581-3bb6-4c00-8da1-01e45c20c236\") " Sep 11 20:06:41.449750 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:41.449648 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b98b581-3bb6-4c00-8da1-01e45c20c236-istio-envoy\") pod \"8b98b581-3bb6-4c00-8da1-01e45c20c236\" (UID: \"8b98b581-3bb6-4c00-8da1-01e45c20c236\") " Sep 11 20:06:41.449750 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:41.449681 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b98b581-3bb6-4c00-8da1-01e45c20c236-istio-token\") pod \"8b98b581-3bb6-4c00-8da1-01e45c20c236\" (UID: \"8b98b581-3bb6-4c00-8da1-01e45c20c236\") " Sep 11 20:06:41.449750 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:41.449732 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b98b581-3bb6-4c00-8da1-01e45c20c236-workload-socket\") pod \"8b98b581-3bb6-4c00-8da1-01e45c20c236\" (UID: \"8b98b581-3bb6-4c00-8da1-01e45c20c236\") " Sep 11 20:06:41.449994 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:41.449771 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b98b581-3bb6-4c00-8da1-01e45c20c236-workload-certs\") pod \"8b98b581-3bb6-4c00-8da1-01e45c20c236\" (UID: \"8b98b581-3bb6-4c00-8da1-01e45c20c236\") " Sep 11 20:06:41.449994 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:41.449795 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b98b581-3bb6-4c00-8da1-01e45c20c236-istio-data\") pod \"8b98b581-3bb6-4c00-8da1-01e45c20c236\" (UID: \"8b98b581-3bb6-4c00-8da1-01e45c20c236\") " Sep 11 20:06:41.449994 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:41.449832 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b98b581-3bb6-4c00-8da1-01e45c20c236-credential-socket\") pod \"8b98b581-3bb6-4c00-8da1-01e45c20c236\" (UID: \"8b98b581-3bb6-4c00-8da1-01e45c20c236\") " Sep 11 20:06:41.450134 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:41.450059 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b98b581-3bb6-4c00-8da1-01e45c20c236-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8b98b581-3bb6-4c00-8da1-01e45c20c236" (UID: "8b98b581-3bb6-4c00-8da1-01e45c20c236"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:06:41.450194 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:41.450176 2546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b98b581-3bb6-4c00-8da1-01e45c20c236-workload-certs\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:06:41.450250 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:41.450180 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b98b581-3bb6-4c00-8da1-01e45c20c236-istio-data" (OuterVolumeSpecName: "istio-data") pod "8b98b581-3bb6-4c00-8da1-01e45c20c236" (UID: "8b98b581-3bb6-4c00-8da1-01e45c20c236"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:06:41.450250 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:41.450070 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b98b581-3bb6-4c00-8da1-01e45c20c236-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8b98b581-3bb6-4c00-8da1-01e45c20c236" (UID: "8b98b581-3bb6-4c00-8da1-01e45c20c236"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:06:41.450353 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:41.450250 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b98b581-3bb6-4c00-8da1-01e45c20c236-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8b98b581-3bb6-4c00-8da1-01e45c20c236" (UID: "8b98b581-3bb6-4c00-8da1-01e45c20c236"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:06:41.450465 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:41.450438 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b98b581-3bb6-4c00-8da1-01e45c20c236-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8b98b581-3bb6-4c00-8da1-01e45c20c236" (UID: "8b98b581-3bb6-4c00-8da1-01e45c20c236"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:06:41.452325 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:41.452300 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b98b581-3bb6-4c00-8da1-01e45c20c236-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8b98b581-3bb6-4c00-8da1-01e45c20c236" (UID: "8b98b581-3bb6-4c00-8da1-01e45c20c236"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 20:06:41.452428 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:41.452318 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b98b581-3bb6-4c00-8da1-01e45c20c236-kube-api-access-5dt88" (OuterVolumeSpecName: "kube-api-access-5dt88") pod "8b98b581-3bb6-4c00-8da1-01e45c20c236" (UID: "8b98b581-3bb6-4c00-8da1-01e45c20c236"). InnerVolumeSpecName "kube-api-access-5dt88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:06:41.452428 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:41.452374 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b98b581-3bb6-4c00-8da1-01e45c20c236-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8b98b581-3bb6-4c00-8da1-01e45c20c236" (UID: "8b98b581-3bb6-4c00-8da1-01e45c20c236"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:06:41.452428 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:41.452387 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b98b581-3bb6-4c00-8da1-01e45c20c236-istio-token" (OuterVolumeSpecName: "istio-token") pod "8b98b581-3bb6-4c00-8da1-01e45c20c236" (UID: "8b98b581-3bb6-4c00-8da1-01e45c20c236"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:06:41.551971 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:41.551927 2546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b98b581-3bb6-4c00-8da1-01e45c20c236-istio-token\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:06:41.551971 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:41.551959 2546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b98b581-3bb6-4c00-8da1-01e45c20c236-workload-socket\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:06:41.551971 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:41.551974 2546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b98b581-3bb6-4c00-8da1-01e45c20c236-istio-data\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:06:41.552262 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:41.551988 2546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b98b581-3bb6-4c00-8da1-01e45c20c236-credential-socket\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:06:41.552262 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:41.552003 2546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b98b581-3bb6-4c00-8da1-01e45c20c236-istio-podinfo\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:06:41.552262 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:41.552015 2546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b98b581-3bb6-4c00-8da1-01e45c20c236-istiod-ca-cert\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:06:41.552262 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:41.552026 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dt88\" (UniqueName: \"kubernetes.io/projected/8b98b581-3bb6-4c00-8da1-01e45c20c236-kube-api-access-5dt88\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:06:41.552262 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:41.552034 2546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b98b581-3bb6-4c00-8da1-01e45c20c236-istio-envoy\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:06:42.094640 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:42.094609 2546 generic.go:356] "Generic (PLEG): container finished" podID="8b98b581-3bb6-4c00-8da1-01e45c20c236" containerID="8a5701c66c4d52f7c15df0a5bf68bb4232e3469f5e845858097bd0905c4deee5" exitCode=0 Sep 11 20:06:42.094810 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:42.094657 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk" Sep 11 20:06:42.094810 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:42.094662 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk" event={"ID":"8b98b581-3bb6-4c00-8da1-01e45c20c236","Type":"ContainerDied","Data":"8a5701c66c4d52f7c15df0a5bf68bb4232e3469f5e845858097bd0905c4deee5"} Sep 11 20:06:42.094810 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:42.094778 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk" event={"ID":"8b98b581-3bb6-4c00-8da1-01e45c20c236","Type":"ContainerDied","Data":"61c14c3d12fe30e9ae4ad9d7b40a22dd201813e14b63375103fed52d2a2997ea"} Sep 11 20:06:42.094810 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:42.094807 2546 scope.go:117] "RemoveContainer" containerID="8a5701c66c4d52f7c15df0a5bf68bb4232e3469f5e845858097bd0905c4deee5" Sep 11 20:06:42.122636 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:42.122611 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk"] Sep 11 20:06:42.135957 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:42.135932 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcnsrqk"] Sep 11 20:06:42.454269 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:42.454185 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b98b581-3bb6-4c00-8da1-01e45c20c236" path="/var/lib/kubelet/pods/8b98b581-3bb6-4c00-8da1-01e45c20c236/volumes" Sep 11 20:06:42.570484 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:42.570019 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:42.625317 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:42.625258 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-xzdbn"] Sep 11 20:06:42.626091 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:42.625809 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b98b581-3bb6-4c00-8da1-01e45c20c236" containerName="istio-proxy" Sep 11 20:06:42.626091 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:42.625831 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b98b581-3bb6-4c00-8da1-01e45c20c236" containerName="istio-proxy" Sep 11 20:06:42.626091 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:42.625903 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b98b581-3bb6-4c00-8da1-01e45c20c236" containerName="istio-proxy" Sep 11 20:06:42.632688 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:42.632658 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xzdbn" Sep 11 20:06:42.634738 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:42.634716 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Sep 11 20:06:42.634860 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:42.634817 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-lz5ln\"" Sep 11 20:06:42.640050 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:42.640023 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-hs68x"] Sep 11 20:06:42.643797 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:42.643689 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hs68x" Sep 11 20:06:42.643942 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:42.643916 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-xzdbn"] Sep 11 20:06:42.645890 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:42.645868 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Sep 11 20:06:42.646248 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:42.646213 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-td4ss\"" Sep 11 20:06:42.664934 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:42.664908 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-hs68x"] Sep 11 20:06:42.673904 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:42.673864 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:42.763497 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:42.763404 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7c00411d-ee5f-4b0f-8c45-2daa2426389d-cert\") pod \"kserve-controller-manager-d6f4b65b8-xzdbn\" (UID: \"7c00411d-ee5f-4b0f-8c45-2daa2426389d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xzdbn" Sep 11 20:06:42.763497 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:42.763478 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4s8p\" (UniqueName: \"kubernetes.io/projected/9c8afe06-4c3c-41dd-9adb-f3cdc51ee68d-kube-api-access-f4s8p\") pod \"llmisvc-controller-manager-777c6bd7d5-hs68x\" (UID: \"9c8afe06-4c3c-41dd-9adb-f3cdc51ee68d\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hs68x" Sep 11 20:06:42.763844 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:42.763632 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d9h8\" (UniqueName: \"kubernetes.io/projected/7c00411d-ee5f-4b0f-8c45-2daa2426389d-kube-api-access-4d9h8\") pod \"kserve-controller-manager-d6f4b65b8-xzdbn\" (UID: \"7c00411d-ee5f-4b0f-8c45-2daa2426389d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xzdbn" Sep 11 20:06:42.763844 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:42.763687 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9c8afe06-4c3c-41dd-9adb-f3cdc51ee68d-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-hs68x\" (UID: \"9c8afe06-4c3c-41dd-9adb-f3cdc51ee68d\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hs68x" Sep 11 20:06:42.864798 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:42.864765 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d9h8\" (UniqueName: \"kubernetes.io/projected/7c00411d-ee5f-4b0f-8c45-2daa2426389d-kube-api-access-4d9h8\") pod \"kserve-controller-manager-d6f4b65b8-xzdbn\" (UID: \"7c00411d-ee5f-4b0f-8c45-2daa2426389d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xzdbn" Sep 11 20:06:42.864798 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:42.864808 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9c8afe06-4c3c-41dd-9adb-f3cdc51ee68d-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-hs68x\" (UID: \"9c8afe06-4c3c-41dd-9adb-f3cdc51ee68d\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hs68x" Sep 11 20:06:42.865032 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:42.864847 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7c00411d-ee5f-4b0f-8c45-2daa2426389d-cert\") pod \"kserve-controller-manager-d6f4b65b8-xzdbn\" (UID: \"7c00411d-ee5f-4b0f-8c45-2daa2426389d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xzdbn" Sep 11 20:06:42.865032 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:42.864881 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4s8p\" (UniqueName: \"kubernetes.io/projected/9c8afe06-4c3c-41dd-9adb-f3cdc51ee68d-kube-api-access-f4s8p\") pod \"llmisvc-controller-manager-777c6bd7d5-hs68x\" (UID: \"9c8afe06-4c3c-41dd-9adb-f3cdc51ee68d\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hs68x" Sep 11 20:06:42.865032 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:06:42.864936 2546 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Sep 11 20:06:42.865032 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:06:42.865010 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c8afe06-4c3c-41dd-9adb-f3cdc51ee68d-cert podName:9c8afe06-4c3c-41dd-9adb-f3cdc51ee68d nodeName:}" failed. No retries permitted until 2026-09-11 20:06:43.364988729 +0000 UTC m=+373.413039051 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/9c8afe06-4c3c-41dd-9adb-f3cdc51ee68d-cert") pod "llmisvc-controller-manager-777c6bd7d5-hs68x" (UID: "9c8afe06-4c3c-41dd-9adb-f3cdc51ee68d") : secret "llmisvc-webhook-server-cert" not found Sep 11 20:06:42.865032 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:06:42.865010 2546 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Sep 11 20:06:42.865294 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:06:42.865073 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c00411d-ee5f-4b0f-8c45-2daa2426389d-cert podName:7c00411d-ee5f-4b0f-8c45-2daa2426389d nodeName:}" failed. No retries permitted until 2026-09-11 20:06:43.365055279 +0000 UTC m=+373.413105605 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/7c00411d-ee5f-4b0f-8c45-2daa2426389d-cert") pod "kserve-controller-manager-d6f4b65b8-xzdbn" (UID: "7c00411d-ee5f-4b0f-8c45-2daa2426389d") : secret "kserve-webhook-server-cert" not found Sep 11 20:06:42.884259 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:42.884224 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d9h8\" (UniqueName: \"kubernetes.io/projected/7c00411d-ee5f-4b0f-8c45-2daa2426389d-kube-api-access-4d9h8\") pod \"kserve-controller-manager-d6f4b65b8-xzdbn\" (UID: \"7c00411d-ee5f-4b0f-8c45-2daa2426389d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xzdbn" Sep 11 20:06:42.884563 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:42.884539 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4s8p\" (UniqueName: \"kubernetes.io/projected/9c8afe06-4c3c-41dd-9adb-f3cdc51ee68d-kube-api-access-f4s8p\") pod \"llmisvc-controller-manager-777c6bd7d5-hs68x\" (UID: \"9c8afe06-4c3c-41dd-9adb-f3cdc51ee68d\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hs68x" Sep 11 20:06:42.954268 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:42.954238 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:42.975084 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:42.974902 2546 scope.go:117] "RemoveContainer" containerID="8a5701c66c4d52f7c15df0a5bf68bb4232e3469f5e845858097bd0905c4deee5" Sep 11 20:06:42.975231 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:06:42.975210 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a5701c66c4d52f7c15df0a5bf68bb4232e3469f5e845858097bd0905c4deee5\": container with ID starting with 8a5701c66c4d52f7c15df0a5bf68bb4232e3469f5e845858097bd0905c4deee5 not found: ID does not exist" containerID="8a5701c66c4d52f7c15df0a5bf68bb4232e3469f5e845858097bd0905c4deee5" Sep 11 20:06:42.975277 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:42.975239 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a5701c66c4d52f7c15df0a5bf68bb4232e3469f5e845858097bd0905c4deee5"} err="failed to get container status \"8a5701c66c4d52f7c15df0a5bf68bb4232e3469f5e845858097bd0905c4deee5\": rpc error: code = NotFound desc = could not find container \"8a5701c66c4d52f7c15df0a5bf68bb4232e3469f5e845858097bd0905c4deee5\": container with ID starting with 8a5701c66c4d52f7c15df0a5bf68bb4232e3469f5e845858097bd0905c4deee5 not found: ID does not exist" Sep 11 20:06:43.032836 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:43.032799 2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 20:06:43.032944 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:43.032870 2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 20:06:43.099183 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:43.099155 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:43.100745 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:43.100720 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pnz6j" event={"ID":"d6aef9ef-d852-445e-9538-7e487cdbbf6f","Type":"ContainerStarted","Data":"554680c4fe7e8d920163219d851f090e14c3213d6bd260fc00459e3fcb6be5df"} Sep 11 20:06:43.100886 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:43.100860 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pnz6j" Sep 11 20:06:43.120716 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:43.120673 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pnz6j" podStartSLOduration=1.520828726 podStartE2EDuration="4.120659289s" totalImagesPullingTime="2.599830563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:06:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:06:43.118932019 +0000 UTC m=+373.166982354" watchObservedRunningTime="2026-09-11 20:06:43.120659289 +0000 UTC m=+373.168709633" Sep 11 20:06:43.369526 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:43.369486 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9c8afe06-4c3c-41dd-9adb-f3cdc51ee68d-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-hs68x\" (UID: \"9c8afe06-4c3c-41dd-9adb-f3cdc51ee68d\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hs68x" Sep 11 20:06:43.369750 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:43.369559 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7c00411d-ee5f-4b0f-8c45-2daa2426389d-cert\") pod \"kserve-controller-manager-d6f4b65b8-xzdbn\" (UID: \"7c00411d-ee5f-4b0f-8c45-2daa2426389d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xzdbn" Sep 11 20:06:43.372636 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:43.372607 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7c00411d-ee5f-4b0f-8c45-2daa2426389d-cert\") pod \"kserve-controller-manager-d6f4b65b8-xzdbn\" (UID: \"7c00411d-ee5f-4b0f-8c45-2daa2426389d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xzdbn" Sep 11 20:06:43.372843 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:43.372815 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9c8afe06-4c3c-41dd-9adb-f3cdc51ee68d-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-hs68x\" (UID: \"9c8afe06-4c3c-41dd-9adb-f3cdc51ee68d\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hs68x" Sep 11 20:06:43.548212 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:43.548177 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xzdbn" Sep 11 20:06:43.559363 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:43.559342 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hs68x" Sep 11 20:06:43.830905 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:43.830863 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-xzdbn"] Sep 11 20:06:43.834692 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:06:43.834661 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c00411d_ee5f_4b0f_8c45_2daa2426389d.slice/crio-ea2c946dc88173a72cbc4edb214db41326e6e16281d9ca2a255781022a818514 WatchSource:0}: Error finding container ea2c946dc88173a72cbc4edb214db41326e6e16281d9ca2a255781022a818514: Status 404 returned error can't find the container with id ea2c946dc88173a72cbc4edb214db41326e6e16281d9ca2a255781022a818514 Sep 11 20:06:43.849276 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:43.849242 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-hs68x"] Sep 11 20:06:43.853129 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:06:43.853104 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9c8afe06_4c3c_41dd_9adb_f3cdc51ee68d.slice/crio-2f4562b133e159943d7dbf5892d4d4eeb63eca2d44e8d50dba608973937d41e7 WatchSource:0}: Error finding container 2f4562b133e159943d7dbf5892d4d4eeb63eca2d44e8d50dba608973937d41e7: Status 404 returned error can't find the container with id 2f4562b133e159943d7dbf5892d4d4eeb63eca2d44e8d50dba608973937d41e7 Sep 11 20:06:43.863132 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:43.863111 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:43.938598 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:43.938516 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:44.108557 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:44.108517 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hs68x" event={"ID":"9c8afe06-4c3c-41dd-9adb-f3cdc51ee68d","Type":"ContainerStarted","Data":"2f4562b133e159943d7dbf5892d4d4eeb63eca2d44e8d50dba608973937d41e7"} Sep 11 20:06:44.109666 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:44.109643 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xzdbn" event={"ID":"7c00411d-ee5f-4b0f-8c45-2daa2426389d","Type":"ContainerStarted","Data":"ea2c946dc88173a72cbc4edb214db41326e6e16281d9ca2a255781022a818514"} Sep 11 20:06:44.111600 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:44.111566 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pnz6j" Sep 11 20:06:47.128324 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:47.128283 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xzdbn" event={"ID":"7c00411d-ee5f-4b0f-8c45-2daa2426389d","Type":"ContainerStarted","Data":"9deee1844c3690c113f5fb0639d69442f200402fba384254d6467b439617b812"} Sep 11 20:06:47.129101 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:47.129075 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xzdbn" Sep 11 20:06:47.148836 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:47.148788 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xzdbn" podStartSLOduration=2.67655418 podStartE2EDuration="5.148768375s" totalImagesPullingTime="2.472214195s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:06:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:06:47.146916341 +0000 UTC m=+377.194966688" watchObservedRunningTime="2026-09-11 20:06:47.148768375 +0000 UTC m=+377.196818721" Sep 11 20:06:48.133834 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:48.133792 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hs68x" event={"ID":"9c8afe06-4c3c-41dd-9adb-f3cdc51ee68d","Type":"ContainerStarted","Data":"471ae0455d54b5a9c220b9f94d6bca56b313fd5b75d60fc3de9062993b1101c9"} Sep 11 20:06:48.134311 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:48.134114 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hs68x" Sep 11 20:06:48.148323 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:48.148261 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hs68x" podStartSLOduration=2.97718843 podStartE2EDuration="6.148244041s" totalImagesPullingTime="3.171055611s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:06:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:06:48.147893157 +0000 UTC m=+378.195943502" watchObservedRunningTime="2026-09-11 20:06:48.148244041 +0000 UTC m=+378.196294385" Sep 11 20:06:48.542901 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:48.542816 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7c4d7c7587-68mg4" Sep 11 20:06:48.542901 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:48.542874 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7c4d7c7587-68mg4" Sep 11 20:06:48.547702 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:48.547678 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7c4d7c7587-68mg4" Sep 11 20:06:49.081709 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:49.081680 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-htbhw" Sep 11 20:06:49.083520 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:49.083499 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-hmfh4" Sep 11 20:06:49.143812 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:49.143779 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7c4d7c7587-68mg4" Sep 11 20:06:49.361704 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:49.361673 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:49.461713 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:06:49.461687 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:07:19.142177 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:07:19.142137 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hs68x" Sep 11 20:07:19.145444 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:07:19.145403 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xzdbn" Sep 11 20:07:33.497763 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:07:33.497725 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:07:33.592606 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:07:33.592561 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:07:47.661151 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:07:47.661100 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:07:47.735512 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:07:47.735485 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:07:48.026004 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:07:48.025899 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:07:48.085877 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:07:48.085849 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:07:49.400703 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:07:49.400667 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:07:49.455698 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:07:49.455666 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:07:51.577628 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:07:51.577597 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:01.119448 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:01.119417 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:01.921515 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:01.921478 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:02.718617 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:02.718564 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:05.020681 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:05.020648 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:05.149100 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:05.149066 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:05.989732 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:05.989696 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-w56sw"] Sep 11 20:08:05.993093 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:05.993077 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-w56sw" Sep 11 20:08:05.996737 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:05.996707 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-vn2ls\"" Sep 11 20:08:05.997442 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:05.997427 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 11 20:08:05.997535 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:05.997433 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 11 20:08:06.020308 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:06.020285 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-w56sw"] Sep 11 20:08:06.063417 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:06.063389 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccxm8\" (UniqueName: \"kubernetes.io/projected/29b04777-0ad3-4b03-a33f-eb4868d23e2f-kube-api-access-ccxm8\") pod \"kuadrant-operator-controller-manager-76f8f686bf-w56sw\" (UID: \"29b04777-0ad3-4b03-a33f-eb4868d23e2f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-w56sw" Sep 11 20:08:06.063777 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:06.063559 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/29b04777-0ad3-4b03-a33f-eb4868d23e2f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-w56sw\" (UID: \"29b04777-0ad3-4b03-a33f-eb4868d23e2f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-w56sw" Sep 11 20:08:06.164214 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:06.164185 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/29b04777-0ad3-4b03-a33f-eb4868d23e2f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-w56sw\" (UID: \"29b04777-0ad3-4b03-a33f-eb4868d23e2f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-w56sw" Sep 11 20:08:06.164407 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:06.164237 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccxm8\" (UniqueName: \"kubernetes.io/projected/29b04777-0ad3-4b03-a33f-eb4868d23e2f-kube-api-access-ccxm8\") pod \"kuadrant-operator-controller-manager-76f8f686bf-w56sw\" (UID: \"29b04777-0ad3-4b03-a33f-eb4868d23e2f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-w56sw" Sep 11 20:08:06.164535 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:06.164518 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/29b04777-0ad3-4b03-a33f-eb4868d23e2f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-w56sw\" (UID: \"29b04777-0ad3-4b03-a33f-eb4868d23e2f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-w56sw" Sep 11 20:08:06.182010 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:06.181980 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:06.186077 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:06.186059 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccxm8\" (UniqueName: \"kubernetes.io/projected/29b04777-0ad3-4b03-a33f-eb4868d23e2f-kube-api-access-ccxm8\") pod \"kuadrant-operator-controller-manager-76f8f686bf-w56sw\" (UID: \"29b04777-0ad3-4b03-a33f-eb4868d23e2f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-w56sw" Sep 11 20:08:06.303236 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:06.303173 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-w56sw" Sep 11 20:08:06.456943 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:06.456904 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:06.553668 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:06.553594 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:06.594004 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:06.593981 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-w56sw"] Sep 11 20:08:06.596232 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:08:06.596207 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29b04777_0ad3_4b03_a33f_eb4868d23e2f.slice/crio-d7488ac003aef15c8d58785738aa22c57eb63d37b1b97be534ce48e6467fb8f8 WatchSource:0}: Error finding container d7488ac003aef15c8d58785738aa22c57eb63d37b1b97be534ce48e6467fb8f8: Status 404 returned error can't find the container with id d7488ac003aef15c8d58785738aa22c57eb63d37b1b97be534ce48e6467fb8f8 Sep 11 20:08:06.660490 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:06.660464 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:07.494425 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:07.494396 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-w56sw" event={"ID":"29b04777-0ad3-4b03-a33f-eb4868d23e2f","Type":"ContainerStarted","Data":"d7488ac003aef15c8d58785738aa22c57eb63d37b1b97be534ce48e6467fb8f8"} Sep 11 20:08:09.159669 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:09.159111 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:10.589022 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:10.588992 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-g72s5"] Sep 11 20:08:10.591991 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:10.591972 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g72s5" Sep 11 20:08:10.594070 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:10.594052 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-9v6m9\"" Sep 11 20:08:10.603814 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:10.603793 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-g72s5"] Sep 11 20:08:10.704117 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:10.704091 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsldh\" (UniqueName: \"kubernetes.io/projected/5b0e1959-0e43-4647-bcd3-1bac20499b51-kube-api-access-rsldh\") pod \"limitador-operator-controller-manager-74495c69f8-g72s5\" (UID: \"5b0e1959-0e43-4647-bcd3-1bac20499b51\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g72s5" Sep 11 20:08:10.805440 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:10.805413 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsldh\" (UniqueName: \"kubernetes.io/projected/5b0e1959-0e43-4647-bcd3-1bac20499b51-kube-api-access-rsldh\") pod \"limitador-operator-controller-manager-74495c69f8-g72s5\" (UID: \"5b0e1959-0e43-4647-bcd3-1bac20499b51\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g72s5" Sep 11 20:08:10.813992 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:10.813964 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsldh\" (UniqueName: \"kubernetes.io/projected/5b0e1959-0e43-4647-bcd3-1bac20499b51-kube-api-access-rsldh\") pod \"limitador-operator-controller-manager-74495c69f8-g72s5\" (UID: \"5b0e1959-0e43-4647-bcd3-1bac20499b51\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g72s5" Sep 11 20:08:10.903136 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:10.903056 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g72s5" Sep 11 20:08:11.147499 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:11.147469 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:11.179136 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:11.179085 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-g72s5"] Sep 11 20:08:11.181179 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:08:11.181157 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b0e1959_0e43_4647_bcd3_1bac20499b51.slice/crio-0775bc79d32c2ca4cbbf3707b90d7fde0e0096dee7affc425eb447410dc5dec4 WatchSource:0}: Error finding container 0775bc79d32c2ca4cbbf3707b90d7fde0e0096dee7affc425eb447410dc5dec4: Status 404 returned error can't find the container with id 0775bc79d32c2ca4cbbf3707b90d7fde0e0096dee7affc425eb447410dc5dec4 Sep 11 20:08:11.374130 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:11.374095 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:11.517758 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:11.517670 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-w56sw" event={"ID":"29b04777-0ad3-4b03-a33f-eb4868d23e2f","Type":"ContainerStarted","Data":"82d5aefcef7c2fd0e0f8cf13b1b71c5d272ad74e164ab499dcc34af982d4babd"} Sep 11 20:08:11.517758 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:11.517751 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-w56sw" Sep 11 20:08:11.519137 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:11.519113 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g72s5" event={"ID":"5b0e1959-0e43-4647-bcd3-1bac20499b51","Type":"ContainerStarted","Data":"0775bc79d32c2ca4cbbf3707b90d7fde0e0096dee7affc425eb447410dc5dec4"} Sep 11 20:08:11.542230 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:11.542188 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-w56sw" podStartSLOduration=2.501540644 podStartE2EDuration="6.542171429s" totalImagesPullingTime="4.040630785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:08:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:08:11.540673065 +0000 UTC m=+461.588723400" watchObservedRunningTime="2026-09-11 20:08:11.542171429 +0000 UTC m=+461.590221773" Sep 11 20:08:13.237219 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:13.237180 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:13.531555 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:13.531481 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g72s5" event={"ID":"5b0e1959-0e43-4647-bcd3-1bac20499b51","Type":"ContainerStarted","Data":"daeef5c9b87aef3d67e6d5bd251a46cfae3d2f9275892a5a3753d491c0d124c4"} Sep 11 20:08:13.531707 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:13.531628 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g72s5" Sep 11 20:08:13.584300 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:13.584247 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g72s5" podStartSLOduration=1.396579018 podStartE2EDuration="3.584229789s" totalImagesPullingTime="2.187650771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:08:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:08:13.582565911 +0000 UTC m=+463.630616254" watchObservedRunningTime="2026-09-11 20:08:13.584229789 +0000 UTC m=+463.632280132" Sep 11 20:08:13.650942 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:13.650910 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:13.830517 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:13.830424 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:15.741562 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:15.741529 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:15.842778 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:15.842749 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:15.942109 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:15.942075 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:16.043228 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:16.043153 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:16.931570 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:16.931534 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:16.989227 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:16.989200 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:22.527437 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:22.527409 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-w56sw" Sep 11 20:08:24.539047 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:24.539019 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g72s5" Sep 11 20:08:24.701965 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:24.701928 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-w56sw"] Sep 11 20:08:24.702169 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:24.702143 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-w56sw" podUID="29b04777-0ad3-4b03-a33f-eb4868d23e2f" containerName="manager" containerID="cri-o://82d5aefcef7c2fd0e0f8cf13b1b71c5d272ad74e164ab499dcc34af982d4babd" gracePeriod=2 Sep 11 20:08:24.725284 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:24.725260 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-w56sw"] Sep 11 20:08:24.746589 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:08:24.746543 2546 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"29b04777-0ad3-4b03-a33f-eb4868d23e2f\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-w56sw\": pods \"kuadrant-operator-controller-manager-76f8f686bf-w56sw\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-w56sw" Sep 11 20:08:24.753538 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:24.753514 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-g72s5"] Sep 11 20:08:24.753707 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:24.753687 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g72s5" podUID="5b0e1959-0e43-4647-bcd3-1bac20499b51" containerName="manager" containerID="cri-o://daeef5c9b87aef3d67e6d5bd251a46cfae3d2f9275892a5a3753d491c0d124c4" gracePeriod=2 Sep 11 20:08:24.756930 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:08:24.756899 2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-w56sw\" is forbidden: User \"system:node:ip-10-0-138-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-205.ec2.internal' and this object" podUID="29b04777-0ad3-4b03-a33f-eb4868d23e2f" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-w56sw" Sep 11 20:08:24.765293 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:24.765270 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jpkml"] Sep 11 20:08:24.765655 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:24.765640 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29b04777-0ad3-4b03-a33f-eb4868d23e2f" containerName="manager" Sep 11 20:08:24.765655 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:24.765655 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="29b04777-0ad3-4b03-a33f-eb4868d23e2f" containerName="manager" Sep 11 20:08:24.765785 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:24.765730 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="29b04777-0ad3-4b03-a33f-eb4868d23e2f" containerName="manager" Sep 11 20:08:24.768785 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:24.768769 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-g72s5"] Sep 11 20:08:24.768870 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:24.768849 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jpkml" Sep 11 20:08:24.792473 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:24.792341 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jpkml"] Sep 11 20:08:24.815846 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:24.815816 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-gdgsc"] Sep 11 20:08:24.816445 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:24.816420 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b0e1959-0e43-4647-bcd3-1bac20499b51" containerName="manager" Sep 11 20:08:24.816445 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:24.816446 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b0e1959-0e43-4647-bcd3-1bac20499b51" containerName="manager" Sep 11 20:08:24.816847 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:24.816593 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b0e1959-0e43-4647-bcd3-1bac20499b51" containerName="manager" Sep 11 20:08:24.818272 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:08:24.818245 2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-w56sw\" is forbidden: User \"system:node:ip-10-0-138-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-205.ec2.internal' and this object" podUID="29b04777-0ad3-4b03-a33f-eb4868d23e2f" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-w56sw" Sep 11 20:08:24.819881 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:24.819865 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-gdgsc" Sep 11 20:08:24.821186 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:08:24.821160 2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-g72s5\" is forbidden: User \"system:node:ip-10-0-138-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-205.ec2.internal' and this object" podUID="5b0e1959-0e43-4647-bcd3-1bac20499b51" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g72s5" Sep 11 20:08:24.826279 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:08:24.826256 2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-w56sw\" is forbidden: User \"system:node:ip-10-0-138-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-205.ec2.internal' and this object" podUID="29b04777-0ad3-4b03-a33f-eb4868d23e2f" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-w56sw" Sep 11 20:08:24.828667 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:08:24.828635 2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-g72s5\" is forbidden: User \"system:node:ip-10-0-138-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-205.ec2.internal' and this object" podUID="5b0e1959-0e43-4647-bcd3-1bac20499b51" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g72s5" Sep 11 20:08:24.829964 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:24.829940 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0bc4e97b-f77c-44a9-be59-d70a97b01a0b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-jpkml\" (UID: \"0bc4e97b-f77c-44a9-be59-d70a97b01a0b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jpkml" Sep 11 20:08:24.830055 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:24.829975 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vflrh\" (UniqueName: \"kubernetes.io/projected/0bc4e97b-f77c-44a9-be59-d70a97b01a0b-kube-api-access-vflrh\") pod \"kuadrant-operator-controller-manager-76f8f686bf-jpkml\" (UID: \"0bc4e97b-f77c-44a9-be59-d70a97b01a0b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jpkml" Sep 11 20:08:24.830055 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:24.830026 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsb5r\" (UniqueName: \"kubernetes.io/projected/34685262-d863-498c-9550-f9a0a20cef98-kube-api-access-tsb5r\") pod \"limitador-operator-controller-manager-74495c69f8-gdgsc\" (UID: \"34685262-d863-498c-9550-f9a0a20cef98\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-gdgsc" Sep 11 20:08:24.841294 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:24.841270 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-gdgsc"] Sep 11 20:08:24.930676 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:24.930646 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0bc4e97b-f77c-44a9-be59-d70a97b01a0b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-jpkml\" (UID: \"0bc4e97b-f77c-44a9-be59-d70a97b01a0b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jpkml" Sep 11 20:08:24.930829 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:24.930688 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vflrh\" (UniqueName: \"kubernetes.io/projected/0bc4e97b-f77c-44a9-be59-d70a97b01a0b-kube-api-access-vflrh\") pod \"kuadrant-operator-controller-manager-76f8f686bf-jpkml\" (UID: \"0bc4e97b-f77c-44a9-be59-d70a97b01a0b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jpkml" Sep 11 20:08:24.930829 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:24.930733 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsb5r\" (UniqueName: \"kubernetes.io/projected/34685262-d863-498c-9550-f9a0a20cef98-kube-api-access-tsb5r\") pod \"limitador-operator-controller-manager-74495c69f8-gdgsc\" (UID: \"34685262-d863-498c-9550-f9a0a20cef98\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-gdgsc" Sep 11 20:08:24.931366 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:24.931338 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0bc4e97b-f77c-44a9-be59-d70a97b01a0b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-jpkml\" (UID: \"0bc4e97b-f77c-44a9-be59-d70a97b01a0b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jpkml" Sep 11 20:08:24.940849 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:24.940810 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsb5r\" (UniqueName: \"kubernetes.io/projected/34685262-d863-498c-9550-f9a0a20cef98-kube-api-access-tsb5r\") pod \"limitador-operator-controller-manager-74495c69f8-gdgsc\" (UID: \"34685262-d863-498c-9550-f9a0a20cef98\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-gdgsc" Sep 11 20:08:24.941390 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:24.941368 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vflrh\" (UniqueName: \"kubernetes.io/projected/0bc4e97b-f77c-44a9-be59-d70a97b01a0b-kube-api-access-vflrh\") pod \"kuadrant-operator-controller-manager-76f8f686bf-jpkml\" (UID: \"0bc4e97b-f77c-44a9-be59-d70a97b01a0b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jpkml" Sep 11 20:08:24.979514 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:24.979493 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-w56sw" Sep 11 20:08:24.982024 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:24.982007 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g72s5" Sep 11 20:08:24.982676 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:08:24.982649 2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-w56sw\" is forbidden: User \"system:node:ip-10-0-138-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-205.ec2.internal' and this object" podUID="29b04777-0ad3-4b03-a33f-eb4868d23e2f" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-w56sw" Sep 11 20:08:24.984796 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:08:24.984769 2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-g72s5\" is forbidden: User \"system:node:ip-10-0-138-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-205.ec2.internal' and this object" podUID="5b0e1959-0e43-4647-bcd3-1bac20499b51" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g72s5" Sep 11 20:08:24.988473 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:08:24.988452 2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-w56sw\" is forbidden: User \"system:node:ip-10-0-138-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-205.ec2.internal' and this object" podUID="29b04777-0ad3-4b03-a33f-eb4868d23e2f" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-w56sw" Sep 11 20:08:24.989903 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:08:24.989884 2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-g72s5\" is forbidden: User \"system:node:ip-10-0-138-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-205.ec2.internal' and this object" podUID="5b0e1959-0e43-4647-bcd3-1bac20499b51" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g72s5" Sep 11 20:08:25.031851 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:25.031829 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ccxm8\" (UniqueName: \"kubernetes.io/projected/29b04777-0ad3-4b03-a33f-eb4868d23e2f-kube-api-access-ccxm8\") pod \"29b04777-0ad3-4b03-a33f-eb4868d23e2f\" (UID: \"29b04777-0ad3-4b03-a33f-eb4868d23e2f\") " Sep 11 20:08:25.031924 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:25.031856 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rsldh\" (UniqueName: \"kubernetes.io/projected/5b0e1959-0e43-4647-bcd3-1bac20499b51-kube-api-access-rsldh\") pod \"5b0e1959-0e43-4647-bcd3-1bac20499b51\" (UID: \"5b0e1959-0e43-4647-bcd3-1bac20499b51\") " Sep 11 20:08:25.031924 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:25.031883 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/29b04777-0ad3-4b03-a33f-eb4868d23e2f-extensions-socket-volume\") pod \"29b04777-0ad3-4b03-a33f-eb4868d23e2f\" (UID: \"29b04777-0ad3-4b03-a33f-eb4868d23e2f\") " Sep 11 20:08:25.032389 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:25.032362 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29b04777-0ad3-4b03-a33f-eb4868d23e2f-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "29b04777-0ad3-4b03-a33f-eb4868d23e2f" (UID: "29b04777-0ad3-4b03-a33f-eb4868d23e2f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:08:25.033880 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:25.033861 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29b04777-0ad3-4b03-a33f-eb4868d23e2f-kube-api-access-ccxm8" (OuterVolumeSpecName: "kube-api-access-ccxm8") pod "29b04777-0ad3-4b03-a33f-eb4868d23e2f" (UID: "29b04777-0ad3-4b03-a33f-eb4868d23e2f"). InnerVolumeSpecName "kube-api-access-ccxm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:08:25.033943 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:25.033897 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b0e1959-0e43-4647-bcd3-1bac20499b51-kube-api-access-rsldh" (OuterVolumeSpecName: "kube-api-access-rsldh") pod "5b0e1959-0e43-4647-bcd3-1bac20499b51" (UID: "5b0e1959-0e43-4647-bcd3-1bac20499b51"). InnerVolumeSpecName "kube-api-access-rsldh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:08:25.132421 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:25.132398 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccxm8\" (UniqueName: \"kubernetes.io/projected/29b04777-0ad3-4b03-a33f-eb4868d23e2f-kube-api-access-ccxm8\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:08:25.132421 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:25.132419 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsldh\" (UniqueName: \"kubernetes.io/projected/5b0e1959-0e43-4647-bcd3-1bac20499b51-kube-api-access-rsldh\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:08:25.132553 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:25.132429 2546 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/29b04777-0ad3-4b03-a33f-eb4868d23e2f-extensions-socket-volume\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:08:25.157314 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:25.157281 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jpkml" Sep 11 20:08:25.164323 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:25.164309 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-gdgsc" Sep 11 20:08:25.436217 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:25.436157 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jpkml"] Sep 11 20:08:25.439069 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:08:25.439037 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bc4e97b_f77c_44a9_be59_d70a97b01a0b.slice/crio-6502db1dcf02c2a1921ac50a68aee0d57d2dd903732294336b22b01349181017 WatchSource:0}: Error finding container 6502db1dcf02c2a1921ac50a68aee0d57d2dd903732294336b22b01349181017: Status 404 returned error can't find the container with id 6502db1dcf02c2a1921ac50a68aee0d57d2dd903732294336b22b01349181017 Sep 11 20:08:25.439829 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:25.439807 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-gdgsc"] Sep 11 20:08:25.441892 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:08:25.441871 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34685262_d863_498c_9550_f9a0a20cef98.slice/crio-0797d89c13f2bfdc61504aa14704c84646ea518b6fa35fe2331315efdf563e37 WatchSource:0}: Error finding container 0797d89c13f2bfdc61504aa14704c84646ea518b6fa35fe2331315efdf563e37: Status 404 returned error can't find the container with id 0797d89c13f2bfdc61504aa14704c84646ea518b6fa35fe2331315efdf563e37 Sep 11 20:08:25.588096 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:25.588066 2546 generic.go:356] "Generic (PLEG): container finished" podID="5b0e1959-0e43-4647-bcd3-1bac20499b51" containerID="daeef5c9b87aef3d67e6d5bd251a46cfae3d2f9275892a5a3753d491c0d124c4" exitCode=0 Sep 11 20:08:25.588507 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:25.588107 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g72s5" Sep 11 20:08:25.588507 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:25.588164 2546 scope.go:117] "RemoveContainer" containerID="daeef5c9b87aef3d67e6d5bd251a46cfae3d2f9275892a5a3753d491c0d124c4" Sep 11 20:08:25.590084 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:25.590056 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-gdgsc" event={"ID":"34685262-d863-498c-9550-f9a0a20cef98","Type":"ContainerStarted","Data":"2222bb569c79cde91ee363b9a1fca3531713bd6996d53f4ac8307a1077be4b42"} Sep 11 20:08:25.590219 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:25.590109 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-gdgsc" event={"ID":"34685262-d863-498c-9550-f9a0a20cef98","Type":"ContainerStarted","Data":"0797d89c13f2bfdc61504aa14704c84646ea518b6fa35fe2331315efdf563e37"} Sep 11 20:08:25.590219 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:25.590150 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-gdgsc" Sep 11 20:08:25.590350 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:08:25.590234 2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-w56sw\" is forbidden: User \"system:node:ip-10-0-138-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-205.ec2.internal' and this object" podUID="29b04777-0ad3-4b03-a33f-eb4868d23e2f" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-w56sw" Sep 11 20:08:25.591955 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:25.591935 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jpkml" event={"ID":"0bc4e97b-f77c-44a9-be59-d70a97b01a0b","Type":"ContainerStarted","Data":"add750af0a798047fa2559fd126fb5718212a30773c706a1b9e61847eb12eee9"} Sep 11 20:08:25.592072 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:25.591978 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jpkml" event={"ID":"0bc4e97b-f77c-44a9-be59-d70a97b01a0b","Type":"ContainerStarted","Data":"6502db1dcf02c2a1921ac50a68aee0d57d2dd903732294336b22b01349181017"} Sep 11 20:08:25.592072 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:25.592009 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jpkml" Sep 11 20:08:25.592504 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:08:25.592475 2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-g72s5\" is forbidden: User \"system:node:ip-10-0-138-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-205.ec2.internal' and this object" podUID="5b0e1959-0e43-4647-bcd3-1bac20499b51" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g72s5" Sep 11 20:08:25.593549 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:25.593524 2546 generic.go:356] "Generic (PLEG): container finished" podID="29b04777-0ad3-4b03-a33f-eb4868d23e2f" containerID="82d5aefcef7c2fd0e0f8cf13b1b71c5d272ad74e164ab499dcc34af982d4babd" exitCode=0 Sep 11 20:08:25.593655 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:25.593566 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-w56sw" Sep 11 20:08:25.594160 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:08:25.594140 2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-w56sw\" is forbidden: User \"system:node:ip-10-0-138-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-205.ec2.internal' and this object" podUID="29b04777-0ad3-4b03-a33f-eb4868d23e2f" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-w56sw" Sep 11 20:08:25.595867 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:08:25.595847 2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-g72s5\" is forbidden: User \"system:node:ip-10-0-138-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-205.ec2.internal' and this object" podUID="5b0e1959-0e43-4647-bcd3-1bac20499b51" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g72s5" Sep 11 20:08:25.597723 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:25.597701 2546 scope.go:117] "RemoveContainer" containerID="daeef5c9b87aef3d67e6d5bd251a46cfae3d2f9275892a5a3753d491c0d124c4" Sep 11 20:08:25.597988 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:08:25.597970 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daeef5c9b87aef3d67e6d5bd251a46cfae3d2f9275892a5a3753d491c0d124c4\": container with ID starting with daeef5c9b87aef3d67e6d5bd251a46cfae3d2f9275892a5a3753d491c0d124c4 not found: ID does not exist" containerID="daeef5c9b87aef3d67e6d5bd251a46cfae3d2f9275892a5a3753d491c0d124c4" Sep 11 20:08:25.598042 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:25.597994 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daeef5c9b87aef3d67e6d5bd251a46cfae3d2f9275892a5a3753d491c0d124c4"} err="failed to get container status \"daeef5c9b87aef3d67e6d5bd251a46cfae3d2f9275892a5a3753d491c0d124c4\": rpc error: code = NotFound desc = could not find container \"daeef5c9b87aef3d67e6d5bd251a46cfae3d2f9275892a5a3753d491c0d124c4\": container with ID starting with daeef5c9b87aef3d67e6d5bd251a46cfae3d2f9275892a5a3753d491c0d124c4 not found: ID does not exist" Sep 11 20:08:25.598042 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:25.598010 2546 scope.go:117] "RemoveContainer" containerID="82d5aefcef7c2fd0e0f8cf13b1b71c5d272ad74e164ab499dcc34af982d4babd" Sep 11 20:08:25.609273 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:25.609251 2546 scope.go:117] "RemoveContainer" containerID="82d5aefcef7c2fd0e0f8cf13b1b71c5d272ad74e164ab499dcc34af982d4babd" Sep 11 20:08:25.609521 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:08:25.609504 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82d5aefcef7c2fd0e0f8cf13b1b71c5d272ad74e164ab499dcc34af982d4babd\": container with ID starting with 82d5aefcef7c2fd0e0f8cf13b1b71c5d272ad74e164ab499dcc34af982d4babd not found: ID does not exist" containerID="82d5aefcef7c2fd0e0f8cf13b1b71c5d272ad74e164ab499dcc34af982d4babd" Sep 11 20:08:25.609596 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:25.609528 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82d5aefcef7c2fd0e0f8cf13b1b71c5d272ad74e164ab499dcc34af982d4babd"} err="failed to get container status \"82d5aefcef7c2fd0e0f8cf13b1b71c5d272ad74e164ab499dcc34af982d4babd\": rpc error: code = NotFound desc = could not find container \"82d5aefcef7c2fd0e0f8cf13b1b71c5d272ad74e164ab499dcc34af982d4babd\": container with ID starting with 82d5aefcef7c2fd0e0f8cf13b1b71c5d272ad74e164ab499dcc34af982d4babd not found: ID does not exist" Sep 11 20:08:25.622933 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:25.622890 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-gdgsc" podStartSLOduration=1.622878478 podStartE2EDuration="1.622878478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:08:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:08:25.621216556 +0000 UTC m=+475.669266899" watchObservedRunningTime="2026-09-11 20:08:25.622878478 +0000 UTC m=+475.670928822" Sep 11 20:08:25.644682 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:25.644634 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jpkml" podStartSLOduration=1.644619612 podStartE2EDuration="1.644619612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:08:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:08:25.643143445 +0000 UTC m=+475.691193789" watchObservedRunningTime="2026-09-11 20:08:25.644619612 +0000 UTC m=+475.692669960" Sep 11 20:08:25.645708 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:08:25.645686 2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-w56sw\" is forbidden: User \"system:node:ip-10-0-138-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-205.ec2.internal' and this object" podUID="29b04777-0ad3-4b03-a33f-eb4868d23e2f" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-w56sw" Sep 11 20:08:25.647612 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:08:25.647568 2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-g72s5\" is forbidden: User \"system:node:ip-10-0-138-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-205.ec2.internal' and this object" podUID="5b0e1959-0e43-4647-bcd3-1bac20499b51" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g72s5" Sep 11 20:08:26.450656 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:26.450627 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29b04777-0ad3-4b03-a33f-eb4868d23e2f" path="/var/lib/kubelet/pods/29b04777-0ad3-4b03-a33f-eb4868d23e2f/volumes" Sep 11 20:08:26.450996 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:26.450982 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b0e1959-0e43-4647-bcd3-1bac20499b51" path="/var/lib/kubelet/pods/5b0e1959-0e43-4647-bcd3-1bac20499b51/volumes" Sep 11 20:08:26.540552 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:26.540519 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:26.646429 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:26.646399 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:27.152127 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:27.152096 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:27.253371 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:27.253342 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:30.930611 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:30.930547 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:31.031657 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:31.031626 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:33.958756 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:33.958724 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:34.059917 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:34.059885 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:36.602928 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:36.602896 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-gdgsc" Sep 11 20:08:36.603331 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:36.602970 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jpkml" Sep 11 20:08:40.732161 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:40.732126 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:40.798947 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:40.798918 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:41.333244 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:41.333212 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:41.395767 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:41.395740 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:41.636645 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:41.636605 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jpkml"] Sep 11 20:08:41.636874 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:41.636850 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jpkml" podUID="0bc4e97b-f77c-44a9-be59-d70a97b01a0b" containerName="manager" containerID="cri-o://add750af0a798047fa2559fd126fb5718212a30773c706a1b9e61847eb12eee9" gracePeriod=10 Sep 11 20:08:41.750388 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:41.750356 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:41.851115 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:41.851083 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:41.880832 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:41.880811 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jpkml" Sep 11 20:08:41.969070 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:41.968992 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vflrh\" (UniqueName: \"kubernetes.io/projected/0bc4e97b-f77c-44a9-be59-d70a97b01a0b-kube-api-access-vflrh\") pod \"0bc4e97b-f77c-44a9-be59-d70a97b01a0b\" (UID: \"0bc4e97b-f77c-44a9-be59-d70a97b01a0b\") " Sep 11 20:08:41.969209 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:41.969077 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0bc4e97b-f77c-44a9-be59-d70a97b01a0b-extensions-socket-volume\") pod \"0bc4e97b-f77c-44a9-be59-d70a97b01a0b\" (UID: \"0bc4e97b-f77c-44a9-be59-d70a97b01a0b\") " Sep 11 20:08:41.969506 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:41.969478 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bc4e97b-f77c-44a9-be59-d70a97b01a0b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "0bc4e97b-f77c-44a9-be59-d70a97b01a0b" (UID: "0bc4e97b-f77c-44a9-be59-d70a97b01a0b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:08:41.970996 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:41.970972 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bc4e97b-f77c-44a9-be59-d70a97b01a0b-kube-api-access-vflrh" (OuterVolumeSpecName: "kube-api-access-vflrh") pod "0bc4e97b-f77c-44a9-be59-d70a97b01a0b" (UID: "0bc4e97b-f77c-44a9-be59-d70a97b01a0b"). InnerVolumeSpecName "kube-api-access-vflrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:08:42.070460 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:42.070420 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vflrh\" (UniqueName: \"kubernetes.io/projected/0bc4e97b-f77c-44a9-be59-d70a97b01a0b-kube-api-access-vflrh\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:08:42.070460 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:42.070457 2546 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0bc4e97b-f77c-44a9-be59-d70a97b01a0b-extensions-socket-volume\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:08:42.685506 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:42.685473 2546 generic.go:356] "Generic (PLEG): container finished" podID="0bc4e97b-f77c-44a9-be59-d70a97b01a0b" containerID="add750af0a798047fa2559fd126fb5718212a30773c706a1b9e61847eb12eee9" exitCode=0 Sep 11 20:08:42.685686 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:42.685523 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jpkml" Sep 11 20:08:42.685686 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:42.685556 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jpkml" event={"ID":"0bc4e97b-f77c-44a9-be59-d70a97b01a0b","Type":"ContainerDied","Data":"add750af0a798047fa2559fd126fb5718212a30773c706a1b9e61847eb12eee9"} Sep 11 20:08:42.685686 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:42.685622 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jpkml" event={"ID":"0bc4e97b-f77c-44a9-be59-d70a97b01a0b","Type":"ContainerDied","Data":"6502db1dcf02c2a1921ac50a68aee0d57d2dd903732294336b22b01349181017"} Sep 11 20:08:42.685686 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:42.685645 2546 scope.go:117] "RemoveContainer" containerID="add750af0a798047fa2559fd126fb5718212a30773c706a1b9e61847eb12eee9" Sep 11 20:08:42.693983 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:42.693963 2546 scope.go:117] "RemoveContainer" containerID="add750af0a798047fa2559fd126fb5718212a30773c706a1b9e61847eb12eee9" Sep 11 20:08:42.694260 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:08:42.694240 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"add750af0a798047fa2559fd126fb5718212a30773c706a1b9e61847eb12eee9\": container with ID starting with add750af0a798047fa2559fd126fb5718212a30773c706a1b9e61847eb12eee9 not found: ID does not exist" containerID="add750af0a798047fa2559fd126fb5718212a30773c706a1b9e61847eb12eee9" Sep 11 20:08:42.694308 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:42.694268 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"add750af0a798047fa2559fd126fb5718212a30773c706a1b9e61847eb12eee9"} err="failed to get container status \"add750af0a798047fa2559fd126fb5718212a30773c706a1b9e61847eb12eee9\": rpc error: code = NotFound desc = could not find container \"add750af0a798047fa2559fd126fb5718212a30773c706a1b9e61847eb12eee9\": container with ID starting with add750af0a798047fa2559fd126fb5718212a30773c706a1b9e61847eb12eee9 not found: ID does not exist" Sep 11 20:08:42.724181 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:42.724129 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jpkml"] Sep 11 20:08:42.738198 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:42.738168 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jpkml"] Sep 11 20:08:44.449984 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:44.449949 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bc4e97b-f77c-44a9-be59-d70a97b01a0b" path="/var/lib/kubelet/pods/0bc4e97b-f77c-44a9-be59-d70a97b01a0b/volumes" Sep 11 20:08:49.739465 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:49.739429 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:49.839310 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:49.839279 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:52.746191 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:52.746156 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:52.850947 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:52.850916 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:55.186287 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:55.186256 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:55.286554 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:55.286523 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:57.969402 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:57.969371 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-r787z"] Sep 11 20:08:57.969950 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:57.969930 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bc4e97b-f77c-44a9-be59-d70a97b01a0b" containerName="manager" Sep 11 20:08:57.970027 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:57.969953 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bc4e97b-f77c-44a9-be59-d70a97b01a0b" containerName="manager" Sep 11 20:08:57.970082 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:57.970056 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bc4e97b-f77c-44a9-be59-d70a97b01a0b" containerName="manager" Sep 11 20:08:57.973169 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:57.973146 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-r787z" Sep 11 20:08:57.975987 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:57.975969 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-4jlr2\"" Sep 11 20:08:57.991123 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:57.991101 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-r787z"] Sep 11 20:08:58.003821 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:58.003792 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:58.097620 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:58.097558 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07f88bd9-1f8b-400f-bf08-e1a1b90c014f-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-r787z\" (UID: \"07f88bd9-1f8b-400f-bf08-e1a1b90c014f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-r787z" Sep 11 20:08:58.097620 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:58.097617 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07f88bd9-1f8b-400f-bf08-e1a1b90c014f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-r787z\" (UID: \"07f88bd9-1f8b-400f-bf08-e1a1b90c014f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-r787z" Sep 11 20:08:58.097811 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:58.097677 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbp9m\" (UniqueName: \"kubernetes.io/projected/07f88bd9-1f8b-400f-bf08-e1a1b90c014f-kube-api-access-sbp9m\") pod \"maas-default-gateway-openshift-default-78bb78ff78-r787z\" (UID: \"07f88bd9-1f8b-400f-bf08-e1a1b90c014f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-r787z" Sep 11 20:08:58.097811 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:58.097721 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07f88bd9-1f8b-400f-bf08-e1a1b90c014f-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-r787z\" (UID: \"07f88bd9-1f8b-400f-bf08-e1a1b90c014f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-r787z" Sep 11 20:08:58.097811 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:58.097736 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07f88bd9-1f8b-400f-bf08-e1a1b90c014f-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-r787z\" (UID: \"07f88bd9-1f8b-400f-bf08-e1a1b90c014f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-r787z" Sep 11 20:08:58.097811 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:58.097756 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07f88bd9-1f8b-400f-bf08-e1a1b90c014f-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-r787z\" (UID: \"07f88bd9-1f8b-400f-bf08-e1a1b90c014f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-r787z" Sep 11 20:08:58.097942 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:58.097844 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07f88bd9-1f8b-400f-bf08-e1a1b90c014f-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-r787z\" (UID: \"07f88bd9-1f8b-400f-bf08-e1a1b90c014f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-r787z" Sep 11 20:08:58.097942 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:58.097877 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07f88bd9-1f8b-400f-bf08-e1a1b90c014f-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-r787z\" (UID: \"07f88bd9-1f8b-400f-bf08-e1a1b90c014f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-r787z" Sep 11 20:08:58.097942 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:58.097903 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07f88bd9-1f8b-400f-bf08-e1a1b90c014f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-r787z\" (UID: \"07f88bd9-1f8b-400f-bf08-e1a1b90c014f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-r787z" Sep 11 20:08:58.198708 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:58.198669 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbp9m\" (UniqueName: \"kubernetes.io/projected/07f88bd9-1f8b-400f-bf08-e1a1b90c014f-kube-api-access-sbp9m\") pod \"maas-default-gateway-openshift-default-78bb78ff78-r787z\" (UID: \"07f88bd9-1f8b-400f-bf08-e1a1b90c014f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-r787z" Sep 11 20:08:58.198885 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:58.198731 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07f88bd9-1f8b-400f-bf08-e1a1b90c014f-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-r787z\" (UID: \"07f88bd9-1f8b-400f-bf08-e1a1b90c014f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-r787z" Sep 11 20:08:58.198885 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:58.198751 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07f88bd9-1f8b-400f-bf08-e1a1b90c014f-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-r787z\" (UID: \"07f88bd9-1f8b-400f-bf08-e1a1b90c014f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-r787z" Sep 11 20:08:58.198885 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:58.198769 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07f88bd9-1f8b-400f-bf08-e1a1b90c014f-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-r787z\" (UID: \"07f88bd9-1f8b-400f-bf08-e1a1b90c014f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-r787z" Sep 11 20:08:58.198885 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:58.198810 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07f88bd9-1f8b-400f-bf08-e1a1b90c014f-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-r787z\" (UID: \"07f88bd9-1f8b-400f-bf08-e1a1b90c014f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-r787z" Sep 11 20:08:58.198885 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:58.198829 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07f88bd9-1f8b-400f-bf08-e1a1b90c014f-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-r787z\" (UID: \"07f88bd9-1f8b-400f-bf08-e1a1b90c014f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-r787z" Sep 11 20:08:58.198885 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:58.198857 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07f88bd9-1f8b-400f-bf08-e1a1b90c014f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-r787z\" (UID: \"07f88bd9-1f8b-400f-bf08-e1a1b90c014f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-r787z" Sep 11 20:08:58.199208 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:58.198897 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07f88bd9-1f8b-400f-bf08-e1a1b90c014f-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-r787z\" (UID: \"07f88bd9-1f8b-400f-bf08-e1a1b90c014f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-r787z" Sep 11 20:08:58.199208 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:58.198932 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07f88bd9-1f8b-400f-bf08-e1a1b90c014f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-r787z\" (UID: \"07f88bd9-1f8b-400f-bf08-e1a1b90c014f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-r787z" Sep 11 20:08:58.199208 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:58.199121 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07f88bd9-1f8b-400f-bf08-e1a1b90c014f-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-r787z\" (UID: \"07f88bd9-1f8b-400f-bf08-e1a1b90c014f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-r787z" Sep 11 20:08:58.199208 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:58.199151 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07f88bd9-1f8b-400f-bf08-e1a1b90c014f-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-r787z\" (UID: \"07f88bd9-1f8b-400f-bf08-e1a1b90c014f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-r787z" Sep 11 20:08:58.199398 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:58.199303 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07f88bd9-1f8b-400f-bf08-e1a1b90c014f-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-r787z\" (UID: \"07f88bd9-1f8b-400f-bf08-e1a1b90c014f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-r787z" Sep 11 20:08:58.199619 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:58.199595 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07f88bd9-1f8b-400f-bf08-e1a1b90c014f-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-r787z\" (UID: \"07f88bd9-1f8b-400f-bf08-e1a1b90c014f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-r787z" Sep 11 20:08:58.199745 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:58.199719 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07f88bd9-1f8b-400f-bf08-e1a1b90c014f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-r787z\" (UID: \"07f88bd9-1f8b-400f-bf08-e1a1b90c014f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-r787z" Sep 11 20:08:58.201186 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:58.201165 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07f88bd9-1f8b-400f-bf08-e1a1b90c014f-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-r787z\" (UID: \"07f88bd9-1f8b-400f-bf08-e1a1b90c014f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-r787z" Sep 11 20:08:58.201433 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:58.201413 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07f88bd9-1f8b-400f-bf08-e1a1b90c014f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-r787z\" (UID: \"07f88bd9-1f8b-400f-bf08-e1a1b90c014f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-r787z" Sep 11 20:08:58.207590 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:58.207554 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbp9m\" (UniqueName: \"kubernetes.io/projected/07f88bd9-1f8b-400f-bf08-e1a1b90c014f-kube-api-access-sbp9m\") pod \"maas-default-gateway-openshift-default-78bb78ff78-r787z\" (UID: \"07f88bd9-1f8b-400f-bf08-e1a1b90c014f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-r787z" Sep 11 20:08:58.207779 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:58.207759 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07f88bd9-1f8b-400f-bf08-e1a1b90c014f-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-r787z\" (UID: \"07f88bd9-1f8b-400f-bf08-e1a1b90c014f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-r787z" Sep 11 20:08:58.283300 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:58.283249 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-r787z" Sep 11 20:08:58.411691 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:58.411656 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:58.562073 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:58.561991 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-r787z"] Sep 11 20:08:58.564263 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:08:58.564234 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07f88bd9_1f8b_400f_bf08_e1a1b90c014f.slice/crio-bfae3a41e86218ae2f470e1c322cb6c5f53ab05af214be320ce67e6617453a7b WatchSource:0}: Error finding container bfae3a41e86218ae2f470e1c322cb6c5f53ab05af214be320ce67e6617453a7b: Status 404 returned error can't find the container with id bfae3a41e86218ae2f470e1c322cb6c5f53ab05af214be320ce67e6617453a7b Sep 11 20:08:58.566343 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:58.566312 2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 20:08:58.566431 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:58.566375 2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 20:08:58.566431 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:58.566419 2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 20:08:58.762097 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:58.762059 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-r787z" event={"ID":"07f88bd9-1f8b-400f-bf08-e1a1b90c014f","Type":"ContainerStarted","Data":"8ae0d7e5887f0cd1af0e810baf1acdcf49d973c1c29080eef7e68331a91ff4e0"} Sep 11 20:08:58.762255 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:58.762125 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-r787z" event={"ID":"07f88bd9-1f8b-400f-bf08-e1a1b90c014f","Type":"ContainerStarted","Data":"bfae3a41e86218ae2f470e1c322cb6c5f53ab05af214be320ce67e6617453a7b"} Sep 11 20:08:58.783117 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:58.783066 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-r787z" podStartSLOduration=1.783046578 podStartE2EDuration="1.783046578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:08:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:08:58.779776854 +0000 UTC m=+508.827827198" watchObservedRunningTime="2026-09-11 20:08:58.783046578 +0000 UTC m=+508.831096923" Sep 11 20:08:59.284255 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:59.284223 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-r787z" Sep 11 20:08:59.288569 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:59.288542 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-r787z" Sep 11 20:08:59.767285 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:59.767253 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-r787z" Sep 11 20:08:59.768265 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:59.768241 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-r787z" Sep 11 20:08:59.833087 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:59.833053 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:59.869478 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:59.869445 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-5m5nf"] Sep 11 20:08:59.873517 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:59.873493 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-5m5nf" Sep 11 20:08:59.876401 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:59.876375 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 11 20:08:59.877105 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:59.877088 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-65m5k\"" Sep 11 20:08:59.896165 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:59.896136 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-5m5nf"] Sep 11 20:08:59.936598 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:59.936558 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-5m5nf"] Sep 11 20:08:59.939480 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:08:59.939458 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:09:00.013913 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:00.013880 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdzgr\" (UniqueName: \"kubernetes.io/projected/da8a7eea-33d7-4ced-a6d7-f07b4edc725e-kube-api-access-bdzgr\") pod \"limitador-limitador-8588749977-5m5nf\" (UID: \"da8a7eea-33d7-4ced-a6d7-f07b4edc725e\") " pod="kuadrant-system/limitador-limitador-8588749977-5m5nf" Sep 11 20:09:00.014054 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:00.013987 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/da8a7eea-33d7-4ced-a6d7-f07b4edc725e-config-file\") pod \"limitador-limitador-8588749977-5m5nf\" (UID: \"da8a7eea-33d7-4ced-a6d7-f07b4edc725e\") " pod="kuadrant-system/limitador-limitador-8588749977-5m5nf" Sep 11 20:09:00.038981 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:00.038899 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:09:00.114999 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:00.114973 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdzgr\" (UniqueName: \"kubernetes.io/projected/da8a7eea-33d7-4ced-a6d7-f07b4edc725e-kube-api-access-bdzgr\") pod \"limitador-limitador-8588749977-5m5nf\" (UID: \"da8a7eea-33d7-4ced-a6d7-f07b4edc725e\") " pod="kuadrant-system/limitador-limitador-8588749977-5m5nf" Sep 11 20:09:00.115158 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:00.115035 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/da8a7eea-33d7-4ced-a6d7-f07b4edc725e-config-file\") pod \"limitador-limitador-8588749977-5m5nf\" (UID: \"da8a7eea-33d7-4ced-a6d7-f07b4edc725e\") " pod="kuadrant-system/limitador-limitador-8588749977-5m5nf" Sep 11 20:09:00.115607 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:00.115563 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/da8a7eea-33d7-4ced-a6d7-f07b4edc725e-config-file\") pod \"limitador-limitador-8588749977-5m5nf\" (UID: \"da8a7eea-33d7-4ced-a6d7-f07b4edc725e\") " pod="kuadrant-system/limitador-limitador-8588749977-5m5nf" Sep 11 20:09:00.123188 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:00.123169 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdzgr\" (UniqueName: \"kubernetes.io/projected/da8a7eea-33d7-4ced-a6d7-f07b4edc725e-kube-api-access-bdzgr\") pod \"limitador-limitador-8588749977-5m5nf\" (UID: \"da8a7eea-33d7-4ced-a6d7-f07b4edc725e\") " pod="kuadrant-system/limitador-limitador-8588749977-5m5nf" Sep 11 20:09:00.187485 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:00.187463 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-5m5nf" Sep 11 20:09:00.458727 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:00.458703 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-5m5nf"] Sep 11 20:09:00.462051 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:09:00.462024 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda8a7eea_33d7_4ced_a6d7_f07b4edc725e.slice/crio-e4f0f18db41fcb944772a0f08186760ccf116cad1a86c0486e8c2224b54c2d64 WatchSource:0}: Error finding container e4f0f18db41fcb944772a0f08186760ccf116cad1a86c0486e8c2224b54c2d64: Status 404 returned error can't find the container with id e4f0f18db41fcb944772a0f08186760ccf116cad1a86c0486e8c2224b54c2d64 Sep 11 20:09:00.650561 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:00.650522 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:09:00.718157 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:00.718089 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:09:00.772958 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:00.772926 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-5m5nf" event={"ID":"da8a7eea-33d7-4ced-a6d7-f07b4edc725e","Type":"ContainerStarted","Data":"e4f0f18db41fcb944772a0f08186760ccf116cad1a86c0486e8c2224b54c2d64"} Sep 11 20:09:02.154673 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:02.154636 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:09:02.255399 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:02.255370 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:09:02.553734 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:02.553638 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:09:02.653984 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:02.653950 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:09:02.755956 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:02.755907 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:09:02.783858 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:02.783824 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-5m5nf" event={"ID":"da8a7eea-33d7-4ced-a6d7-f07b4edc725e","Type":"ContainerStarted","Data":"4443deb6f353c420fdea58322c4069d8a5c2377b49a0d6f5147a26c61725a50f"} Sep 11 20:09:02.784004 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:02.783926 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-5m5nf" Sep 11 20:09:02.801842 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:02.801800 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-5m5nf" podStartSLOduration=2.450280381 podStartE2EDuration="3.801787006s" totalImagesPullingTime="1.351506625s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:08:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:09:02.8001567 +0000 UTC m=+512.848207044" watchObservedRunningTime="2026-09-11 20:09:02.801787006 +0000 UTC m=+512.849837352" Sep 11 20:09:02.856056 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:02.856035 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:09:09.987772 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:09.987739 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:09:10.055102 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:10.055074 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:09:11.557494 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:11.557463 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:09:11.658085 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:11.658051 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:09:13.789824 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:13.789794 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-5m5nf" Sep 11 20:09:13.805774 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:13.805741 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-5m5nf"] Sep 11 20:09:13.837648 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:13.837612 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-5m5nf" podUID="da8a7eea-33d7-4ced-a6d7-f07b4edc725e" containerName="limitador" containerID="cri-o://4443deb6f353c420fdea58322c4069d8a5c2377b49a0d6f5147a26c61725a50f" gracePeriod=30 Sep 11 20:09:14.009904 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:14.009857 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:09:14.073824 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:14.073744 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:09:14.382667 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:14.382643 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-5m5nf" Sep 11 20:09:14.534235 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:14.534199 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/da8a7eea-33d7-4ced-a6d7-f07b4edc725e-config-file\") pod \"da8a7eea-33d7-4ced-a6d7-f07b4edc725e\" (UID: \"da8a7eea-33d7-4ced-a6d7-f07b4edc725e\") " Sep 11 20:09:14.534426 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:14.534322 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bdzgr\" (UniqueName: \"kubernetes.io/projected/da8a7eea-33d7-4ced-a6d7-f07b4edc725e-kube-api-access-bdzgr\") pod \"da8a7eea-33d7-4ced-a6d7-f07b4edc725e\" (UID: \"da8a7eea-33d7-4ced-a6d7-f07b4edc725e\") " Sep 11 20:09:14.534650 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:14.534624 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da8a7eea-33d7-4ced-a6d7-f07b4edc725e-config-file" (OuterVolumeSpecName: "config-file") pod "da8a7eea-33d7-4ced-a6d7-f07b4edc725e" (UID: "da8a7eea-33d7-4ced-a6d7-f07b4edc725e"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:09:14.536322 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:14.536299 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da8a7eea-33d7-4ced-a6d7-f07b4edc725e-kube-api-access-bdzgr" (OuterVolumeSpecName: "kube-api-access-bdzgr") pod "da8a7eea-33d7-4ced-a6d7-f07b4edc725e" (UID: "da8a7eea-33d7-4ced-a6d7-f07b4edc725e"). InnerVolumeSpecName "kube-api-access-bdzgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:09:14.589100 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:14.589069 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:09:14.636070 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:14.635995 2546 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/da8a7eea-33d7-4ced-a6d7-f07b4edc725e-config-file\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:09:14.636070 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:14.636022 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdzgr\" (UniqueName: \"kubernetes.io/projected/da8a7eea-33d7-4ced-a6d7-f07b4edc725e-kube-api-access-bdzgr\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:09:14.779983 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:14.779953 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:09:14.842878 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:14.842840 2546 generic.go:356] "Generic (PLEG): container finished" podID="da8a7eea-33d7-4ced-a6d7-f07b4edc725e" containerID="4443deb6f353c420fdea58322c4069d8a5c2377b49a0d6f5147a26c61725a50f" exitCode=0 Sep 11 20:09:14.843275 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:14.842886 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-5m5nf" event={"ID":"da8a7eea-33d7-4ced-a6d7-f07b4edc725e","Type":"ContainerDied","Data":"4443deb6f353c420fdea58322c4069d8a5c2377b49a0d6f5147a26c61725a50f"} Sep 11 20:09:14.843275 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:14.842901 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-5m5nf" Sep 11 20:09:14.843275 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:14.842939 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-5m5nf" event={"ID":"da8a7eea-33d7-4ced-a6d7-f07b4edc725e","Type":"ContainerDied","Data":"e4f0f18db41fcb944772a0f08186760ccf116cad1a86c0486e8c2224b54c2d64"} Sep 11 20:09:14.843275 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:14.842968 2546 scope.go:117] "RemoveContainer" containerID="4443deb6f353c420fdea58322c4069d8a5c2377b49a0d6f5147a26c61725a50f" Sep 11 20:09:14.851491 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:14.851475 2546 scope.go:117] "RemoveContainer" containerID="4443deb6f353c420fdea58322c4069d8a5c2377b49a0d6f5147a26c61725a50f" Sep 11 20:09:14.851780 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:09:14.851752 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4443deb6f353c420fdea58322c4069d8a5c2377b49a0d6f5147a26c61725a50f\": container with ID starting with 4443deb6f353c420fdea58322c4069d8a5c2377b49a0d6f5147a26c61725a50f not found: ID does not exist" containerID="4443deb6f353c420fdea58322c4069d8a5c2377b49a0d6f5147a26c61725a50f" Sep 11 20:09:14.851853 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:14.851790 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4443deb6f353c420fdea58322c4069d8a5c2377b49a0d6f5147a26c61725a50f"} err="failed to get container status \"4443deb6f353c420fdea58322c4069d8a5c2377b49a0d6f5147a26c61725a50f\": rpc error: code = NotFound desc = could not find container \"4443deb6f353c420fdea58322c4069d8a5c2377b49a0d6f5147a26c61725a50f\": container with ID starting with 4443deb6f353c420fdea58322c4069d8a5c2377b49a0d6f5147a26c61725a50f not found: ID does not exist" Sep 11 20:09:14.863485 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:14.863462 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-5m5nf"] Sep 11 20:09:14.866804 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:14.866785 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-5m5nf"] Sep 11 20:09:15.838672 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:15.838637 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:09:15.940942 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:15.940907 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:09:16.252278 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:16.252194 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-wjc6h"] Sep 11 20:09:16.252774 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:16.252758 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da8a7eea-33d7-4ced-a6d7-f07b4edc725e" containerName="limitador" Sep 11 20:09:16.252835 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:16.252778 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="da8a7eea-33d7-4ced-a6d7-f07b4edc725e" containerName="limitador" Sep 11 20:09:16.252889 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:16.252879 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="da8a7eea-33d7-4ced-a6d7-f07b4edc725e" containerName="limitador" Sep 11 20:09:16.257357 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:16.257340 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wjc6h" Sep 11 20:09:16.259403 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:16.259385 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 20:09:16.259498 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:16.259399 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-8vddv\"" Sep 11 20:09:16.259866 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:16.259851 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 11 20:09:16.259910 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:16.259892 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Sep 11 20:09:16.267151 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:16.267129 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-wjc6h"] Sep 11 20:09:16.291461 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:16.291439 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:09:16.352321 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:16.352283 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhqk7\" (UniqueName: \"kubernetes.io/projected/d9051b97-427f-482e-9d53-7163fc50cf95-kube-api-access-bhqk7\") pod \"postgres-69fb4594cf-wjc6h\" (UID: \"d9051b97-427f-482e-9d53-7163fc50cf95\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wjc6h" Sep 11 20:09:16.352431 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:16.352347 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d9051b97-427f-482e-9d53-7163fc50cf95-data\") pod \"postgres-69fb4594cf-wjc6h\" (UID: \"d9051b97-427f-482e-9d53-7163fc50cf95\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wjc6h" Sep 11 20:09:16.450214 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:16.450182 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da8a7eea-33d7-4ced-a6d7-f07b4edc725e" path="/var/lib/kubelet/pods/da8a7eea-33d7-4ced-a6d7-f07b4edc725e/volumes" Sep 11 20:09:16.453420 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:16.453404 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d9051b97-427f-482e-9d53-7163fc50cf95-data\") pod \"postgres-69fb4594cf-wjc6h\" (UID: \"d9051b97-427f-482e-9d53-7163fc50cf95\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wjc6h" Sep 11 20:09:16.453492 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:16.453476 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhqk7\" (UniqueName: \"kubernetes.io/projected/d9051b97-427f-482e-9d53-7163fc50cf95-kube-api-access-bhqk7\") pod \"postgres-69fb4594cf-wjc6h\" (UID: \"d9051b97-427f-482e-9d53-7163fc50cf95\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wjc6h" Sep 11 20:09:16.453760 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:16.453745 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d9051b97-427f-482e-9d53-7163fc50cf95-data\") pod \"postgres-69fb4594cf-wjc6h\" (UID: \"d9051b97-427f-482e-9d53-7163fc50cf95\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wjc6h" Sep 11 20:09:16.461476 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:16.461451 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhqk7\" (UniqueName: \"kubernetes.io/projected/d9051b97-427f-482e-9d53-7163fc50cf95-kube-api-access-bhqk7\") pod \"postgres-69fb4594cf-wjc6h\" (UID: \"d9051b97-427f-482e-9d53-7163fc50cf95\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wjc6h" Sep 11 20:09:16.568082 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:16.568025 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wjc6h" Sep 11 20:09:16.836796 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:16.836677 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-wjc6h"] Sep 11 20:09:16.839020 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:09:16.838984 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9051b97_427f_482e_9d53_7163fc50cf95.slice/crio-cae705fccd63b1d785246b6a321c24c719656f427c97e255dc37eec0802e3409 WatchSource:0}: Error finding container cae705fccd63b1d785246b6a321c24c719656f427c97e255dc37eec0802e3409: Status 404 returned error can't find the container with id cae705fccd63b1d785246b6a321c24c719656f427c97e255dc37eec0802e3409 Sep 11 20:09:16.856518 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:16.856491 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wjc6h" event={"ID":"d9051b97-427f-482e-9d53-7163fc50cf95","Type":"ContainerStarted","Data":"cae705fccd63b1d785246b6a321c24c719656f427c97e255dc37eec0802e3409"} Sep 11 20:09:17.010552 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:17.010518 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:09:17.085049 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:17.085020 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:09:21.884786 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:21.884740 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wjc6h" event={"ID":"d9051b97-427f-482e-9d53-7163fc50cf95","Type":"ContainerStarted","Data":"82afc01d45f30159ba35c2d00808065ca68d7795cb07e60251e6b5d2a3753078"} Sep 11 20:09:21.885167 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:21.884877 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wjc6h" Sep 11 20:09:21.899612 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:21.899551 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wjc6h" podStartSLOduration=1.054438183 podStartE2EDuration="5.899538291s" totalImagesPullingTime="4.845100108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:09:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:09:21.897896194 +0000 UTC m=+531.945946538" watchObservedRunningTime="2026-09-11 20:09:21.899538291 +0000 UTC m=+531.947588634" Sep 11 20:09:27.917151 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:27.917076 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wjc6h" Sep 11 20:09:28.219204 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:28.219117 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:09:28.347150 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:28.347119 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:09:36.503615 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:36.503583 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:09:36.534562 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:36.534533 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6c76df795b-mcjnk"] Sep 11 20:09:36.555224 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:36.555190 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6c76df795b-mcjnk"] Sep 11 20:09:36.555374 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:36.555283 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c76df795b-mcjnk" Sep 11 20:09:36.557520 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:36.557495 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 11 20:09:36.557520 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:36.557514 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 11 20:09:36.557520 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:36.557527 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 11 20:09:36.557783 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:36.557641 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-2grdg\"" Sep 11 20:09:36.575098 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:36.575076 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:09:36.619272 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:36.619241 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfwtl\" (UniqueName: \"kubernetes.io/projected/f08854de-940e-4f22-ac25-fa55659c0bca-kube-api-access-tfwtl\") pod \"maas-controller-6c76df795b-mcjnk\" (UID: \"f08854de-940e-4f22-ac25-fa55659c0bca\") " pod="opendatahub/maas-controller-6c76df795b-mcjnk" Sep 11 20:09:36.619391 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:36.619297 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f08854de-940e-4f22-ac25-fa55659c0bca-webhook-cert\") pod \"maas-controller-6c76df795b-mcjnk\" (UID: \"f08854de-940e-4f22-ac25-fa55659c0bca\") " pod="opendatahub/maas-controller-6c76df795b-mcjnk" Sep 11 20:09:36.619391 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:36.619380 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f08854de-940e-4f22-ac25-fa55659c0bca-metrics-certs\") pod \"maas-controller-6c76df795b-mcjnk\" (UID: \"f08854de-940e-4f22-ac25-fa55659c0bca\") " pod="opendatahub/maas-controller-6c76df795b-mcjnk" Sep 11 20:09:36.627288 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:36.627263 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:09:36.688189 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:36.688166 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-57fc56fd56-pp52g"] Sep 11 20:09:36.691797 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:36.691782 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57fc56fd56-pp52g" Sep 11 20:09:36.703339 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:36.703319 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57fc56fd56-pp52g"] Sep 11 20:09:36.720241 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:36.720218 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfwtl\" (UniqueName: \"kubernetes.io/projected/f08854de-940e-4f22-ac25-fa55659c0bca-kube-api-access-tfwtl\") pod \"maas-controller-6c76df795b-mcjnk\" (UID: \"f08854de-940e-4f22-ac25-fa55659c0bca\") " pod="opendatahub/maas-controller-6c76df795b-mcjnk" Sep 11 20:09:36.720342 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:36.720247 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4798791b-9a9d-490f-9885-b3db73dfb831-webhook-cert\") pod \"maas-controller-57fc56fd56-pp52g\" (UID: \"4798791b-9a9d-490f-9885-b3db73dfb831\") " pod="opendatahub/maas-controller-57fc56fd56-pp52g" Sep 11 20:09:36.720395 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:36.720372 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f08854de-940e-4f22-ac25-fa55659c0bca-webhook-cert\") pod \"maas-controller-6c76df795b-mcjnk\" (UID: \"f08854de-940e-4f22-ac25-fa55659c0bca\") " pod="opendatahub/maas-controller-6c76df795b-mcjnk" Sep 11 20:09:36.720438 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:36.720421 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4798791b-9a9d-490f-9885-b3db73dfb831-metrics-certs\") pod \"maas-controller-57fc56fd56-pp52g\" (UID: \"4798791b-9a9d-490f-9885-b3db73dfb831\") " pod="opendatahub/maas-controller-57fc56fd56-pp52g" Sep 11 20:09:36.720483 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:36.720446 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtbzd\" (UniqueName: \"kubernetes.io/projected/4798791b-9a9d-490f-9885-b3db73dfb831-kube-api-access-gtbzd\") pod \"maas-controller-57fc56fd56-pp52g\" (UID: \"4798791b-9a9d-490f-9885-b3db73dfb831\") " pod="opendatahub/maas-controller-57fc56fd56-pp52g" Sep 11 20:09:36.720483 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:36.720472 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f08854de-940e-4f22-ac25-fa55659c0bca-metrics-certs\") pod \"maas-controller-6c76df795b-mcjnk\" (UID: \"f08854de-940e-4f22-ac25-fa55659c0bca\") " pod="opendatahub/maas-controller-6c76df795b-mcjnk" Sep 11 20:09:36.720600 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:09:36.720524 2546 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Sep 11 20:09:36.720600 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:09:36.720542 2546 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found Sep 11 20:09:36.720689 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:09:36.720607 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f08854de-940e-4f22-ac25-fa55659c0bca-webhook-cert podName:f08854de-940e-4f22-ac25-fa55659c0bca nodeName:}" failed. No retries permitted until 2026-09-11 20:09:37.220588181 +0000 UTC m=+547.268638525 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/f08854de-940e-4f22-ac25-fa55659c0bca-webhook-cert") pod "maas-controller-6c76df795b-mcjnk" (UID: "f08854de-940e-4f22-ac25-fa55659c0bca") : secret "maas-controller-webhook-cert" not found Sep 11 20:09:36.720689 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:09:36.720626 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f08854de-940e-4f22-ac25-fa55659c0bca-metrics-certs podName:f08854de-940e-4f22-ac25-fa55659c0bca nodeName:}" failed. No retries permitted until 2026-09-11 20:09:37.220617129 +0000 UTC m=+547.268667454 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f08854de-940e-4f22-ac25-fa55659c0bca-metrics-certs") pod "maas-controller-6c76df795b-mcjnk" (UID: "f08854de-940e-4f22-ac25-fa55659c0bca") : secret "maas-controller-metrics-tls" not found Sep 11 20:09:36.728338 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:36.728314 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfwtl\" (UniqueName: \"kubernetes.io/projected/f08854de-940e-4f22-ac25-fa55659c0bca-kube-api-access-tfwtl\") pod \"maas-controller-6c76df795b-mcjnk\" (UID: \"f08854de-940e-4f22-ac25-fa55659c0bca\") " pod="opendatahub/maas-controller-6c76df795b-mcjnk" Sep 11 20:09:36.821268 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:36.821202 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4798791b-9a9d-490f-9885-b3db73dfb831-metrics-certs\") pod \"maas-controller-57fc56fd56-pp52g\" (UID: \"4798791b-9a9d-490f-9885-b3db73dfb831\") " pod="opendatahub/maas-controller-57fc56fd56-pp52g" Sep 11 20:09:36.821268 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:36.821232 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtbzd\" (UniqueName: \"kubernetes.io/projected/4798791b-9a9d-490f-9885-b3db73dfb831-kube-api-access-gtbzd\") pod \"maas-controller-57fc56fd56-pp52g\" (UID: \"4798791b-9a9d-490f-9885-b3db73dfb831\") " pod="opendatahub/maas-controller-57fc56fd56-pp52g" Sep 11 20:09:36.821419 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:36.821331 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4798791b-9a9d-490f-9885-b3db73dfb831-webhook-cert\") pod \"maas-controller-57fc56fd56-pp52g\" (UID: \"4798791b-9a9d-490f-9885-b3db73dfb831\") " pod="opendatahub/maas-controller-57fc56fd56-pp52g" Sep 11 20:09:36.821419 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:09:36.821347 2546 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found Sep 11 20:09:36.821419 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:09:36.821401 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4798791b-9a9d-490f-9885-b3db73dfb831-metrics-certs podName:4798791b-9a9d-490f-9885-b3db73dfb831 nodeName:}" failed. No retries permitted until 2026-09-11 20:09:37.321384589 +0000 UTC m=+547.369434920 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4798791b-9a9d-490f-9885-b3db73dfb831-metrics-certs") pod "maas-controller-57fc56fd56-pp52g" (UID: "4798791b-9a9d-490f-9885-b3db73dfb831") : secret "maas-controller-metrics-tls" not found Sep 11 20:09:36.821539 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:09:36.821438 2546 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Sep 11 20:09:36.821539 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:09:36.821477 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4798791b-9a9d-490f-9885-b3db73dfb831-webhook-cert podName:4798791b-9a9d-490f-9885-b3db73dfb831 nodeName:}" failed. No retries permitted until 2026-09-11 20:09:37.321468415 +0000 UTC m=+547.369518742 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/4798791b-9a9d-490f-9885-b3db73dfb831-webhook-cert") pod "maas-controller-57fc56fd56-pp52g" (UID: "4798791b-9a9d-490f-9885-b3db73dfb831") : secret "maas-controller-webhook-cert" not found Sep 11 20:09:36.829550 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:36.829528 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtbzd\" (UniqueName: \"kubernetes.io/projected/4798791b-9a9d-490f-9885-b3db73dfb831-kube-api-access-gtbzd\") pod \"maas-controller-57fc56fd56-pp52g\" (UID: \"4798791b-9a9d-490f-9885-b3db73dfb831\") " pod="opendatahub/maas-controller-57fc56fd56-pp52g" Sep 11 20:09:36.910645 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:36.910609 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:09:37.022262 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:37.022232 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:09:37.224547 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:37.224515 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f08854de-940e-4f22-ac25-fa55659c0bca-webhook-cert\") pod \"maas-controller-6c76df795b-mcjnk\" (UID: \"f08854de-940e-4f22-ac25-fa55659c0bca\") " pod="opendatahub/maas-controller-6c76df795b-mcjnk" Sep 11 20:09:37.224719 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:37.224569 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f08854de-940e-4f22-ac25-fa55659c0bca-metrics-certs\") pod \"maas-controller-6c76df795b-mcjnk\" (UID: \"f08854de-940e-4f22-ac25-fa55659c0bca\") " pod="opendatahub/maas-controller-6c76df795b-mcjnk" Sep 11 20:09:37.226902 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:37.226869 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f08854de-940e-4f22-ac25-fa55659c0bca-webhook-cert\") pod \"maas-controller-6c76df795b-mcjnk\" (UID: \"f08854de-940e-4f22-ac25-fa55659c0bca\") " pod="opendatahub/maas-controller-6c76df795b-mcjnk" Sep 11 20:09:37.226902 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:37.226885 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f08854de-940e-4f22-ac25-fa55659c0bca-metrics-certs\") pod \"maas-controller-6c76df795b-mcjnk\" (UID: \"f08854de-940e-4f22-ac25-fa55659c0bca\") " pod="opendatahub/maas-controller-6c76df795b-mcjnk" Sep 11 20:09:37.325280 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:37.325248 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4798791b-9a9d-490f-9885-b3db73dfb831-metrics-certs\") pod \"maas-controller-57fc56fd56-pp52g\" (UID: \"4798791b-9a9d-490f-9885-b3db73dfb831\") " pod="opendatahub/maas-controller-57fc56fd56-pp52g" Sep 11 20:09:37.325419 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:37.325342 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4798791b-9a9d-490f-9885-b3db73dfb831-webhook-cert\") pod \"maas-controller-57fc56fd56-pp52g\" (UID: \"4798791b-9a9d-490f-9885-b3db73dfb831\") " pod="opendatahub/maas-controller-57fc56fd56-pp52g" Sep 11 20:09:37.327403 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:37.327381 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4798791b-9a9d-490f-9885-b3db73dfb831-metrics-certs\") pod \"maas-controller-57fc56fd56-pp52g\" (UID: \"4798791b-9a9d-490f-9885-b3db73dfb831\") " pod="opendatahub/maas-controller-57fc56fd56-pp52g" Sep 11 20:09:37.327543 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:37.327525 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4798791b-9a9d-490f-9885-b3db73dfb831-webhook-cert\") pod \"maas-controller-57fc56fd56-pp52g\" (UID: \"4798791b-9a9d-490f-9885-b3db73dfb831\") " pod="opendatahub/maas-controller-57fc56fd56-pp52g" Sep 11 20:09:37.466244 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:37.466220 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c76df795b-mcjnk" Sep 11 20:09:37.601412 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:37.601379 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57fc56fd56-pp52g" Sep 11 20:09:37.736051 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:37.736026 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6c76df795b-mcjnk"] Sep 11 20:09:37.737563 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:09:37.737535 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf08854de_940e_4f22_ac25_fa55659c0bca.slice/crio-a80a1d6dcae1403374dc4afd3306936a759073129a15639f4625f6fb9866529e WatchSource:0}: Error finding container a80a1d6dcae1403374dc4afd3306936a759073129a15639f4625f6fb9866529e: Status 404 returned error can't find the container with id a80a1d6dcae1403374dc4afd3306936a759073129a15639f4625f6fb9866529e Sep 11 20:09:37.872717 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:37.872690 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57fc56fd56-pp52g"] Sep 11 20:09:37.874531 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:09:37.874501 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4798791b_9a9d_490f_9885_b3db73dfb831.slice/crio-c5f4a7eca758831a55217714e5f8d3e17676060f88911a8d0ed7540fe4fbe5d3 WatchSource:0}: Error finding container c5f4a7eca758831a55217714e5f8d3e17676060f88911a8d0ed7540fe4fbe5d3: Status 404 returned error can't find the container with id c5f4a7eca758831a55217714e5f8d3e17676060f88911a8d0ed7540fe4fbe5d3 Sep 11 20:09:37.963031 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:37.962994 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57fc56fd56-pp52g" event={"ID":"4798791b-9a9d-490f-9885-b3db73dfb831","Type":"ContainerStarted","Data":"c5f4a7eca758831a55217714e5f8d3e17676060f88911a8d0ed7540fe4fbe5d3"} Sep 11 20:09:37.964070 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:37.964049 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c76df795b-mcjnk" event={"ID":"f08854de-940e-4f22-ac25-fa55659c0bca","Type":"ContainerStarted","Data":"a80a1d6dcae1403374dc4afd3306936a759073129a15639f4625f6fb9866529e"} Sep 11 20:09:40.500295 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:40.500263 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:09:40.596092 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:40.596065 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:09:40.985189 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:40.985099 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c76df795b-mcjnk" event={"ID":"f08854de-940e-4f22-ac25-fa55659c0bca","Type":"ContainerStarted","Data":"bc1d5dce94dd8087d4345e1a27fda153750a2bdd15be9617507ad4cd29f2dc32"} Sep 11 20:09:40.985189 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:40.985158 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6c76df795b-mcjnk" Sep 11 20:09:40.986639 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:40.986618 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57fc56fd56-pp52g" event={"ID":"4798791b-9a9d-490f-9885-b3db73dfb831","Type":"ContainerStarted","Data":"42e336c5e20297a297926082a20b85521bf72025ed1171185378287ae16974f7"} Sep 11 20:09:40.986753 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:40.986734 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-57fc56fd56-pp52g" Sep 11 20:09:41.005683 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:41.005634 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6c76df795b-mcjnk" podStartSLOduration=2.402072898 podStartE2EDuration="5.0056195s" totalImagesPullingTime="2.603546602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:09:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:09:41.0031162 +0000 UTC m=+551.051166544" watchObservedRunningTime="2026-09-11 20:09:41.0056195 +0000 UTC m=+551.053669843" Sep 11 20:09:41.046457 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:41.046397 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-57fc56fd56-pp52g" podStartSLOduration=2.590136996 podStartE2EDuration="5.046379884s" totalImagesPullingTime="2.456242888s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:09:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:09:41.045556802 +0000 UTC m=+551.093607140" watchObservedRunningTime="2026-09-11 20:09:41.046379884 +0000 UTC m=+551.094430231" Sep 11 20:09:51.993398 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:51.993362 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-57fc56fd56-pp52g" Sep 11 20:09:51.993794 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:51.993457 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6c76df795b-mcjnk" Sep 11 20:09:52.050427 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:52.050384 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6c76df795b-mcjnk"] Sep 11 20:09:52.050624 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:52.050597 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6c76df795b-mcjnk" podUID="f08854de-940e-4f22-ac25-fa55659c0bca" containerName="manager" containerID="cri-o://bc1d5dce94dd8087d4345e1a27fda153750a2bdd15be9617507ad4cd29f2dc32" gracePeriod=30 Sep 11 20:09:52.300987 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:52.300964 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c76df795b-mcjnk" Sep 11 20:09:52.362990 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:52.362962 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f08854de-940e-4f22-ac25-fa55659c0bca-webhook-cert\") pod \"f08854de-940e-4f22-ac25-fa55659c0bca\" (UID: \"f08854de-940e-4f22-ac25-fa55659c0bca\") " Sep 11 20:09:52.363133 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:52.363066 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f08854de-940e-4f22-ac25-fa55659c0bca-metrics-certs\") pod \"f08854de-940e-4f22-ac25-fa55659c0bca\" (UID: \"f08854de-940e-4f22-ac25-fa55659c0bca\") " Sep 11 20:09:52.363133 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:52.363110 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tfwtl\" (UniqueName: \"kubernetes.io/projected/f08854de-940e-4f22-ac25-fa55659c0bca-kube-api-access-tfwtl\") pod \"f08854de-940e-4f22-ac25-fa55659c0bca\" (UID: \"f08854de-940e-4f22-ac25-fa55659c0bca\") " Sep 11 20:09:52.365148 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:52.365118 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f08854de-940e-4f22-ac25-fa55659c0bca-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f08854de-940e-4f22-ac25-fa55659c0bca" (UID: "f08854de-940e-4f22-ac25-fa55659c0bca"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 20:09:52.365268 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:52.365174 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f08854de-940e-4f22-ac25-fa55659c0bca-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "f08854de-940e-4f22-ac25-fa55659c0bca" (UID: "f08854de-940e-4f22-ac25-fa55659c0bca"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 20:09:52.365268 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:52.365180 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f08854de-940e-4f22-ac25-fa55659c0bca-kube-api-access-tfwtl" (OuterVolumeSpecName: "kube-api-access-tfwtl") pod "f08854de-940e-4f22-ac25-fa55659c0bca" (UID: "f08854de-940e-4f22-ac25-fa55659c0bca"). InnerVolumeSpecName "kube-api-access-tfwtl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:09:52.463907 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:52.463885 2546 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f08854de-940e-4f22-ac25-fa55659c0bca-webhook-cert\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:09:52.463907 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:52.463907 2546 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f08854de-940e-4f22-ac25-fa55659c0bca-metrics-certs\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:09:52.464057 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:52.463916 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfwtl\" (UniqueName: \"kubernetes.io/projected/f08854de-940e-4f22-ac25-fa55659c0bca-kube-api-access-tfwtl\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:09:53.044457 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:53.044421 2546 generic.go:356] "Generic (PLEG): container finished" podID="f08854de-940e-4f22-ac25-fa55659c0bca" containerID="bc1d5dce94dd8087d4345e1a27fda153750a2bdd15be9617507ad4cd29f2dc32" exitCode=0 Sep 11 20:09:53.045014 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:53.044462 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c76df795b-mcjnk" Sep 11 20:09:53.045014 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:53.044498 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c76df795b-mcjnk" event={"ID":"f08854de-940e-4f22-ac25-fa55659c0bca","Type":"ContainerDied","Data":"bc1d5dce94dd8087d4345e1a27fda153750a2bdd15be9617507ad4cd29f2dc32"} Sep 11 20:09:53.045014 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:53.044551 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c76df795b-mcjnk" event={"ID":"f08854de-940e-4f22-ac25-fa55659c0bca","Type":"ContainerDied","Data":"a80a1d6dcae1403374dc4afd3306936a759073129a15639f4625f6fb9866529e"} Sep 11 20:09:53.045014 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:53.044584 2546 scope.go:117] "RemoveContainer" containerID="bc1d5dce94dd8087d4345e1a27fda153750a2bdd15be9617507ad4cd29f2dc32" Sep 11 20:09:53.055734 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:53.055498 2546 scope.go:117] "RemoveContainer" containerID="bc1d5dce94dd8087d4345e1a27fda153750a2bdd15be9617507ad4cd29f2dc32" Sep 11 20:09:53.055961 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:09:53.055943 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc1d5dce94dd8087d4345e1a27fda153750a2bdd15be9617507ad4cd29f2dc32\": container with ID starting with bc1d5dce94dd8087d4345e1a27fda153750a2bdd15be9617507ad4cd29f2dc32 not found: ID does not exist" containerID="bc1d5dce94dd8087d4345e1a27fda153750a2bdd15be9617507ad4cd29f2dc32" Sep 11 20:09:53.056018 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:53.055972 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc1d5dce94dd8087d4345e1a27fda153750a2bdd15be9617507ad4cd29f2dc32"} err="failed to get container status \"bc1d5dce94dd8087d4345e1a27fda153750a2bdd15be9617507ad4cd29f2dc32\": rpc error: code = NotFound desc = could not find container \"bc1d5dce94dd8087d4345e1a27fda153750a2bdd15be9617507ad4cd29f2dc32\": container with ID starting with bc1d5dce94dd8087d4345e1a27fda153750a2bdd15be9617507ad4cd29f2dc32 not found: ID does not exist" Sep 11 20:09:53.063032 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:53.063008 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6c76df795b-mcjnk"] Sep 11 20:09:53.068128 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:53.068107 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6c76df795b-mcjnk"] Sep 11 20:09:54.449679 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:09:54.449646 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f08854de-940e-4f22-ac25-fa55659c0bca" path="/var/lib/kubelet/pods/f08854de-940e-4f22-ac25-fa55659c0bca/volumes" Sep 11 20:10:12.201820 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:12.201783 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:10:12.303199 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:12.303169 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:10:13.595682 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:13.595652 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:10:13.671642 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:13.671608 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:10:13.675402 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:13.675374 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-85cff974b6-78c9p"] Sep 11 20:10:13.675873 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:13.675854 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f08854de-940e-4f22-ac25-fa55659c0bca" containerName="manager" Sep 11 20:10:13.675978 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:13.675875 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="f08854de-940e-4f22-ac25-fa55659c0bca" containerName="manager" Sep 11 20:10:13.676041 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:13.675981 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="f08854de-940e-4f22-ac25-fa55659c0bca" containerName="manager" Sep 11 20:10:13.679849 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:13.679831 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-85cff974b6-78c9p" Sep 11 20:10:13.682112 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:13.682092 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 11 20:10:13.682200 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:13.682120 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 11 20:10:13.682249 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:13.682201 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vrkb7\"" Sep 11 20:10:13.694959 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:13.694938 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-85cff974b6-78c9p"] Sep 11 20:10:13.748739 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:13.748707 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62df1741-005c-49a4-8166-faf72dd73a39-maas-api-tls\") pod \"maas-api-85cff974b6-78c9p\" (UID: \"62df1741-005c-49a4-8166-faf72dd73a39\") " pod="odh-ai-gateway-infra/maas-api-85cff974b6-78c9p" Sep 11 20:10:13.748903 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:13.748756 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62df1741-005c-49a4-8166-faf72dd73a39-openshift-service-ca\") pod \"maas-api-85cff974b6-78c9p\" (UID: \"62df1741-005c-49a4-8166-faf72dd73a39\") " pod="odh-ai-gateway-infra/maas-api-85cff974b6-78c9p" Sep 11 20:10:13.748903 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:13.748834 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62df1741-005c-49a4-8166-faf72dd73a39-metrics-certs\") pod \"maas-api-85cff974b6-78c9p\" (UID: \"62df1741-005c-49a4-8166-faf72dd73a39\") " pod="odh-ai-gateway-infra/maas-api-85cff974b6-78c9p" Sep 11 20:10:13.748903 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:13.748861 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8222\" (UniqueName: \"kubernetes.io/projected/62df1741-005c-49a4-8166-faf72dd73a39-kube-api-access-n8222\") pod \"maas-api-85cff974b6-78c9p\" (UID: \"62df1741-005c-49a4-8166-faf72dd73a39\") " pod="odh-ai-gateway-infra/maas-api-85cff974b6-78c9p" Sep 11 20:10:13.765832 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:13.765801 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:10:13.851047 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:13.850544 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62df1741-005c-49a4-8166-faf72dd73a39-maas-api-tls\") pod \"maas-api-85cff974b6-78c9p\" (UID: \"62df1741-005c-49a4-8166-faf72dd73a39\") " pod="odh-ai-gateway-infra/maas-api-85cff974b6-78c9p" Sep 11 20:10:13.851047 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:13.850668 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62df1741-005c-49a4-8166-faf72dd73a39-openshift-service-ca\") pod \"maas-api-85cff974b6-78c9p\" (UID: \"62df1741-005c-49a4-8166-faf72dd73a39\") " pod="odh-ai-gateway-infra/maas-api-85cff974b6-78c9p" Sep 11 20:10:13.851047 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:13.850765 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62df1741-005c-49a4-8166-faf72dd73a39-metrics-certs\") pod \"maas-api-85cff974b6-78c9p\" (UID: \"62df1741-005c-49a4-8166-faf72dd73a39\") " pod="odh-ai-gateway-infra/maas-api-85cff974b6-78c9p" Sep 11 20:10:13.851047 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:13.850794 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8222\" (UniqueName: \"kubernetes.io/projected/62df1741-005c-49a4-8166-faf72dd73a39-kube-api-access-n8222\") pod \"maas-api-85cff974b6-78c9p\" (UID: \"62df1741-005c-49a4-8166-faf72dd73a39\") " pod="odh-ai-gateway-infra/maas-api-85cff974b6-78c9p" Sep 11 20:10:13.851867 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:13.851833 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62df1741-005c-49a4-8166-faf72dd73a39-openshift-service-ca\") pod \"maas-api-85cff974b6-78c9p\" (UID: \"62df1741-005c-49a4-8166-faf72dd73a39\") " pod="odh-ai-gateway-infra/maas-api-85cff974b6-78c9p" Sep 11 20:10:13.853780 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:13.853756 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62df1741-005c-49a4-8166-faf72dd73a39-maas-api-tls\") pod \"maas-api-85cff974b6-78c9p\" (UID: \"62df1741-005c-49a4-8166-faf72dd73a39\") " pod="odh-ai-gateway-infra/maas-api-85cff974b6-78c9p" Sep 11 20:10:13.854231 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:13.854215 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62df1741-005c-49a4-8166-faf72dd73a39-metrics-certs\") pod \"maas-api-85cff974b6-78c9p\" (UID: \"62df1741-005c-49a4-8166-faf72dd73a39\") " pod="odh-ai-gateway-infra/maas-api-85cff974b6-78c9p" Sep 11 20:10:13.858293 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:13.858268 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8222\" (UniqueName: \"kubernetes.io/projected/62df1741-005c-49a4-8166-faf72dd73a39-kube-api-access-n8222\") pod \"maas-api-85cff974b6-78c9p\" (UID: \"62df1741-005c-49a4-8166-faf72dd73a39\") " pod="odh-ai-gateway-infra/maas-api-85cff974b6-78c9p" Sep 11 20:10:13.866632 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:13.866607 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:10:13.989634 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:13.989598 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-85cff974b6-78c9p" Sep 11 20:10:14.092148 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:14.092116 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:10:14.191805 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:14.191732 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:10:14.198855 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:14.198830 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-hvcj7"] Sep 11 20:10:14.201722 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:14.201707 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-hvcj7" Sep 11 20:10:14.204087 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:14.204062 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 11 20:10:14.204308 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:14.204294 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-blgh8\"" Sep 11 20:10:14.214519 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:14.214496 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-hvcj7"] Sep 11 20:10:14.253923 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:14.253884 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk6v6\" (UniqueName: \"kubernetes.io/projected/1b46934e-d75e-4dd9-8bdc-3329269b451b-kube-api-access-dk6v6\") pod \"payload-pre-processing-59fbc47bf8-hvcj7\" (UID: \"1b46934e-d75e-4dd9-8bdc-3329269b451b\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-hvcj7" Sep 11 20:10:14.254099 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:14.254014 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b46934e-d75e-4dd9-8bdc-3329269b451b-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-hvcj7\" (UID: \"1b46934e-d75e-4dd9-8bdc-3329269b451b\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-hvcj7" Sep 11 20:10:14.268918 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:14.268892 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-85cff974b6-78c9p"] Sep 11 20:10:14.271343 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:10:14.271321 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62df1741_005c_49a4_8166_faf72dd73a39.slice/crio-860a1aa092ed3bae475e905b340635e22d8049e1290c5d49cb679dca47fa1bf7 WatchSource:0}: Error finding container 860a1aa092ed3bae475e905b340635e22d8049e1290c5d49cb679dca47fa1bf7: Status 404 returned error can't find the container with id 860a1aa092ed3bae475e905b340635e22d8049e1290c5d49cb679dca47fa1bf7 Sep 11 20:10:14.354510 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:14.354477 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b46934e-d75e-4dd9-8bdc-3329269b451b-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-hvcj7\" (UID: \"1b46934e-d75e-4dd9-8bdc-3329269b451b\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-hvcj7" Sep 11 20:10:14.354705 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:14.354559 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk6v6\" (UniqueName: \"kubernetes.io/projected/1b46934e-d75e-4dd9-8bdc-3329269b451b-kube-api-access-dk6v6\") pod \"payload-pre-processing-59fbc47bf8-hvcj7\" (UID: \"1b46934e-d75e-4dd9-8bdc-3329269b451b\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-hvcj7" Sep 11 20:10:14.355210 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:14.355185 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b46934e-d75e-4dd9-8bdc-3329269b451b-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-hvcj7\" (UID: \"1b46934e-d75e-4dd9-8bdc-3329269b451b\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-hvcj7" Sep 11 20:10:14.366584 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:14.366548 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk6v6\" (UniqueName: \"kubernetes.io/projected/1b46934e-d75e-4dd9-8bdc-3329269b451b-kube-api-access-dk6v6\") pod \"payload-pre-processing-59fbc47bf8-hvcj7\" (UID: \"1b46934e-d75e-4dd9-8bdc-3329269b451b\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-hvcj7" Sep 11 20:10:14.513608 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:14.513498 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-hvcj7" Sep 11 20:10:14.783002 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:14.782977 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-hvcj7"] Sep 11 20:10:14.785417 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:10:14.785388 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b46934e_d75e_4dd9_8bdc_3329269b451b.slice/crio-1fd790a9d7ee575908e65813677f004bbfd99079b350e2c4659b190021165cc9 WatchSource:0}: Error finding container 1fd790a9d7ee575908e65813677f004bbfd99079b350e2c4659b190021165cc9: Status 404 returned error can't find the container with id 1fd790a9d7ee575908e65813677f004bbfd99079b350e2c4659b190021165cc9 Sep 11 20:10:15.155600 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:15.154667 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-hvcj7" event={"ID":"1b46934e-d75e-4dd9-8bdc-3329269b451b","Type":"ContainerStarted","Data":"1fd790a9d7ee575908e65813677f004bbfd99079b350e2c4659b190021165cc9"} Sep 11 20:10:15.158008 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:15.157970 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-85cff974b6-78c9p" event={"ID":"62df1741-005c-49a4-8166-faf72dd73a39","Type":"ContainerStarted","Data":"860a1aa092ed3bae475e905b340635e22d8049e1290c5d49cb679dca47fa1bf7"} Sep 11 20:10:17.170977 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:17.170940 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-hvcj7" event={"ID":"1b46934e-d75e-4dd9-8bdc-3329269b451b","Type":"ContainerStarted","Data":"5cd27b689236129d6c26b147920dd9570b1557c5d19c484c3ba20d581aaf0acb"} Sep 11 20:10:17.171416 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:17.171141 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-hvcj7" Sep 11 20:10:17.172516 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:17.172492 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-85cff974b6-78c9p" event={"ID":"62df1741-005c-49a4-8166-faf72dd73a39","Type":"ContainerStarted","Data":"d53c179acfc4825d0a6ec283986a1f99c69cd8c9de1889564a09d71881d6777a"} Sep 11 20:10:17.172656 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:17.172537 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-85cff974b6-78c9p" Sep 11 20:10:17.189379 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:17.189336 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-hvcj7" podStartSLOduration=1.399371614 podStartE2EDuration="3.189322931s" totalImagesPullingTime="1.789951317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:10:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:10:17.187295241 +0000 UTC m=+587.235345584" watchObservedRunningTime="2026-09-11 20:10:17.189322931 +0000 UTC m=+587.237373275" Sep 11 20:10:17.206342 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:17.206303 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-85cff974b6-78c9p" podStartSLOduration=1.9019144080000001 podStartE2EDuration="4.20628895s" totalImagesPullingTime="2.304374542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:10:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:10:17.204750382 +0000 UTC m=+587.252800747" watchObservedRunningTime="2026-09-11 20:10:17.20628895 +0000 UTC m=+587.254339343" Sep 11 20:10:21.482858 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:21.482802 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:10:22.503798 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:22.503770 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:10:23.184872 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:23.184840 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-85cff974b6-78c9p" Sep 11 20:10:24.180567 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:24.180537 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:10:24.217386 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:24.217353 2546 generic.go:356] "Generic (PLEG): container finished" podID="07f88bd9-1f8b-400f-bf08-e1a1b90c014f" containerID="8ae0d7e5887f0cd1af0e810baf1acdcf49d973c1c29080eef7e68331a91ff4e0" exitCode=0 Sep 11 20:10:24.217600 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:24.217428 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-r787z" event={"ID":"07f88bd9-1f8b-400f-bf08-e1a1b90c014f","Type":"ContainerDied","Data":"8ae0d7e5887f0cd1af0e810baf1acdcf49d973c1c29080eef7e68331a91ff4e0"} Sep 11 20:10:24.218024 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:24.218002 2546 scope.go:117] "RemoveContainer" containerID="8ae0d7e5887f0cd1af0e810baf1acdcf49d973c1c29080eef7e68331a91ff4e0" Sep 11 20:10:24.219224 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:24.219195 2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 20:10:24.219282 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:24.219259 2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 20:10:24.219322 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:24.219290 2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 20:10:24.275518 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:24.275492 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:10:25.224424 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:25.224392 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-r787z" event={"ID":"07f88bd9-1f8b-400f-bf08-e1a1b90c014f","Type":"ContainerStarted","Data":"fa899d21d1f29f6cda6c976ba17859441b38d7c34b5799c7e208adb9273706fa"} Sep 11 20:10:25.283876 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:25.283845 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-r787z" Sep 11 20:10:25.284845 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:25.284828 2546 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-r787z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Sep 11 20:10:25.284909 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:25.284869 2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-r787z" podUID="07f88bd9-1f8b-400f-bf08-e1a1b90c014f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Sep 11 20:10:25.681730 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:25.681695 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:10:26.283862 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:26.283829 2546 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-r787z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Sep 11 20:10:26.284244 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:26.283879 2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-r787z" podUID="07f88bd9-1f8b-400f-bf08-e1a1b90c014f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Sep 11 20:10:27.284337 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:27.284301 2546 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-r787z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Sep 11 20:10:27.284775 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:27.284354 2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-r787z" podUID="07f88bd9-1f8b-400f-bf08-e1a1b90c014f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Sep 11 20:10:28.179755 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:28.179724 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-hvcj7" Sep 11 20:10:28.283727 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:28.283695 2546 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-r787z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Sep 11 20:10:28.283887 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:28.283755 2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-r787z" podUID="07f88bd9-1f8b-400f-bf08-e1a1b90c014f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Sep 11 20:10:29.284029 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:29.283995 2546 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-r787z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Sep 11 20:10:29.284409 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:29.284049 2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-r787z" podUID="07f88bd9-1f8b-400f-bf08-e1a1b90c014f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Sep 11 20:10:29.767630 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:29.767593 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-r787z" Sep 11 20:10:30.284592 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:30.284535 2546 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-r787z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Sep 11 20:10:30.284999 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:30.284605 2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-r787z" podUID="07f88bd9-1f8b-400f-bf08-e1a1b90c014f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Sep 11 20:10:31.287499 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:31.287467 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-r787z" Sep 11 20:10:31.288363 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:31.288342 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-r787z" Sep 11 20:10:38.086539 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:38.086508 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wxg77"] Sep 11 20:10:38.089338 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:38.089321 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wxg77" Sep 11 20:10:38.092236 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:38.092215 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-j8965\"" Sep 11 20:10:38.092352 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:38.092233 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Sep 11 20:10:38.092352 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:38.092215 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 11 20:10:38.092352 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:38.092271 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 11 20:10:38.099721 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:38.099698 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wxg77"] Sep 11 20:10:38.111200 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:38.111175 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:10:38.166776 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:38.166746 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cfbc7f5-c1ef-4ea4-9087-5b619572d396-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wxg77\" (UID: \"9cfbc7f5-c1ef-4ea4-9087-5b619572d396\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wxg77" Sep 11 20:10:38.166964 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:38.166797 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8sgj\" (UniqueName: \"kubernetes.io/projected/9cfbc7f5-c1ef-4ea4-9087-5b619572d396-kube-api-access-c8sgj\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wxg77\" (UID: \"9cfbc7f5-c1ef-4ea4-9087-5b619572d396\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wxg77" Sep 11 20:10:38.166964 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:38.166817 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cfbc7f5-c1ef-4ea4-9087-5b619572d396-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wxg77\" (UID: \"9cfbc7f5-c1ef-4ea4-9087-5b619572d396\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wxg77" Sep 11 20:10:38.166964 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:38.166864 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cfbc7f5-c1ef-4ea4-9087-5b619572d396-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wxg77\" (UID: \"9cfbc7f5-c1ef-4ea4-9087-5b619572d396\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wxg77" Sep 11 20:10:38.166964 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:38.166936 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cfbc7f5-c1ef-4ea4-9087-5b619572d396-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wxg77\" (UID: \"9cfbc7f5-c1ef-4ea4-9087-5b619572d396\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wxg77" Sep 11 20:10:38.167100 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:38.167017 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cfbc7f5-c1ef-4ea4-9087-5b619572d396-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wxg77\" (UID: \"9cfbc7f5-c1ef-4ea4-9087-5b619572d396\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wxg77" Sep 11 20:10:38.268176 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:38.268144 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cfbc7f5-c1ef-4ea4-9087-5b619572d396-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wxg77\" (UID: \"9cfbc7f5-c1ef-4ea4-9087-5b619572d396\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wxg77" Sep 11 20:10:38.268176 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:38.268187 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cfbc7f5-c1ef-4ea4-9087-5b619572d396-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wxg77\" (UID: \"9cfbc7f5-c1ef-4ea4-9087-5b619572d396\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wxg77" Sep 11 20:10:38.268438 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:38.268220 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8sgj\" (UniqueName: \"kubernetes.io/projected/9cfbc7f5-c1ef-4ea4-9087-5b619572d396-kube-api-access-c8sgj\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wxg77\" (UID: \"9cfbc7f5-c1ef-4ea4-9087-5b619572d396\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wxg77" Sep 11 20:10:38.268438 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:38.268322 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cfbc7f5-c1ef-4ea4-9087-5b619572d396-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wxg77\" (UID: \"9cfbc7f5-c1ef-4ea4-9087-5b619572d396\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wxg77" Sep 11 20:10:38.268438 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:38.268359 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cfbc7f5-c1ef-4ea4-9087-5b619572d396-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wxg77\" (UID: \"9cfbc7f5-c1ef-4ea4-9087-5b619572d396\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wxg77" Sep 11 20:10:38.268438 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:38.268410 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cfbc7f5-c1ef-4ea4-9087-5b619572d396-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wxg77\" (UID: \"9cfbc7f5-c1ef-4ea4-9087-5b619572d396\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wxg77" Sep 11 20:10:38.268802 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:38.268771 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cfbc7f5-c1ef-4ea4-9087-5b619572d396-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wxg77\" (UID: \"9cfbc7f5-c1ef-4ea4-9087-5b619572d396\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wxg77" Sep 11 20:10:38.268876 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:38.268836 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cfbc7f5-c1ef-4ea4-9087-5b619572d396-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wxg77\" (UID: \"9cfbc7f5-c1ef-4ea4-9087-5b619572d396\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wxg77" Sep 11 20:10:38.268876 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:38.268853 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cfbc7f5-c1ef-4ea4-9087-5b619572d396-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wxg77\" (UID: \"9cfbc7f5-c1ef-4ea4-9087-5b619572d396\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wxg77" Sep 11 20:10:38.270681 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:38.270655 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cfbc7f5-c1ef-4ea4-9087-5b619572d396-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wxg77\" (UID: \"9cfbc7f5-c1ef-4ea4-9087-5b619572d396\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wxg77" Sep 11 20:10:38.270810 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:38.270788 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cfbc7f5-c1ef-4ea4-9087-5b619572d396-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wxg77\" (UID: \"9cfbc7f5-c1ef-4ea4-9087-5b619572d396\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wxg77" Sep 11 20:10:38.274755 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:38.274729 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8sgj\" (UniqueName: \"kubernetes.io/projected/9cfbc7f5-c1ef-4ea4-9087-5b619572d396-kube-api-access-c8sgj\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wxg77\" (UID: \"9cfbc7f5-c1ef-4ea4-9087-5b619572d396\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wxg77" Sep 11 20:10:38.400380 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:38.400349 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wxg77" Sep 11 20:10:38.674034 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:38.673999 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wxg77"] Sep 11 20:10:38.675906 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:10:38.675879 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cfbc7f5_c1ef_4ea4_9087_5b619572d396.slice/crio-6d176302228ba79b937178495ea89833cc569fde79e79f212e3486e80d70c744 WatchSource:0}: Error finding container 6d176302228ba79b937178495ea89833cc569fde79e79f212e3486e80d70c744: Status 404 returned error can't find the container with id 6d176302228ba79b937178495ea89833cc569fde79e79f212e3486e80d70c744 Sep 11 20:10:38.677288 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:38.677269 2546 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 20:10:39.310814 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:39.310779 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wxg77" event={"ID":"9cfbc7f5-c1ef-4ea4-9087-5b619572d396","Type":"ContainerStarted","Data":"6d176302228ba79b937178495ea89833cc569fde79e79f212e3486e80d70c744"} Sep 11 20:10:40.317378 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:40.317343 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wxg77" event={"ID":"9cfbc7f5-c1ef-4ea4-9087-5b619572d396","Type":"ContainerStarted","Data":"1f9c13c8827a64f041a883146caae79c93ebeed77901723ea49135e0108fd538"} Sep 11 20:10:40.335937 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:40.335888 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wxg77" podStartSLOduration=1.189931 podStartE2EDuration="2.33587311s" totalImagesPullingTime="1.14594211s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:10:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:10:40.334334427 +0000 UTC m=+610.382384771" watchObservedRunningTime="2026-09-11 20:10:40.33587311 +0000 UTC m=+610.383923496" Sep 11 20:10:46.916414 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:46.916355 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:10:48.400867 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:48.400833 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wxg77" Sep 11 20:10:48.401251 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:48.400947 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wxg77" Sep 11 20:10:48.413294 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:48.413275 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wxg77" Sep 11 20:10:49.374255 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:49.374220 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wxg77" Sep 11 20:10:55.884145 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:55.884065 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:10:59.785709 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:59.785650 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-vq928"] Sep 11 20:10:59.789730 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:59.789708 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vq928" Sep 11 20:10:59.792124 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:59.792101 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Sep 11 20:10:59.801752 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:59.801728 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-vq928"] Sep 11 20:10:59.828526 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:59.828498 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:10:59.960915 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:59.960879 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aad104bb-dc8a-48f5-9ff6-33c6e98102f7-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vq928\" (UID: \"aad104bb-dc8a-48f5-9ff6-33c6e98102f7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vq928" Sep 11 20:10:59.961093 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:59.960932 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aad104bb-dc8a-48f5-9ff6-33c6e98102f7-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vq928\" (UID: \"aad104bb-dc8a-48f5-9ff6-33c6e98102f7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vq928" Sep 11 20:10:59.961093 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:59.960969 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x7pc\" (UniqueName: \"kubernetes.io/projected/aad104bb-dc8a-48f5-9ff6-33c6e98102f7-kube-api-access-2x7pc\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vq928\" (UID: \"aad104bb-dc8a-48f5-9ff6-33c6e98102f7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vq928" Sep 11 20:10:59.961184 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:59.961099 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aad104bb-dc8a-48f5-9ff6-33c6e98102f7-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vq928\" (UID: \"aad104bb-dc8a-48f5-9ff6-33c6e98102f7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vq928" Sep 11 20:10:59.961184 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:59.961149 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aad104bb-dc8a-48f5-9ff6-33c6e98102f7-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vq928\" (UID: \"aad104bb-dc8a-48f5-9ff6-33c6e98102f7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vq928" Sep 11 20:10:59.961281 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:10:59.961184 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aad104bb-dc8a-48f5-9ff6-33c6e98102f7-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vq928\" (UID: \"aad104bb-dc8a-48f5-9ff6-33c6e98102f7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vq928" Sep 11 20:11:00.061865 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:11:00.061778 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aad104bb-dc8a-48f5-9ff6-33c6e98102f7-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vq928\" (UID: \"aad104bb-dc8a-48f5-9ff6-33c6e98102f7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vq928" Sep 11 20:11:00.061865 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:11:00.061833 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aad104bb-dc8a-48f5-9ff6-33c6e98102f7-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vq928\" (UID: \"aad104bb-dc8a-48f5-9ff6-33c6e98102f7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vq928" Sep 11 20:11:00.062076 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:11:00.061866 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aad104bb-dc8a-48f5-9ff6-33c6e98102f7-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vq928\" (UID: \"aad104bb-dc8a-48f5-9ff6-33c6e98102f7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vq928" Sep 11 20:11:00.062076 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:11:00.061916 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aad104bb-dc8a-48f5-9ff6-33c6e98102f7-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vq928\" (UID: \"aad104bb-dc8a-48f5-9ff6-33c6e98102f7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vq928" Sep 11 20:11:00.062076 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:11:00.061954 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aad104bb-dc8a-48f5-9ff6-33c6e98102f7-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vq928\" (UID: \"aad104bb-dc8a-48f5-9ff6-33c6e98102f7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vq928" Sep 11 20:11:00.062076 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:11:00.061994 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x7pc\" (UniqueName: \"kubernetes.io/projected/aad104bb-dc8a-48f5-9ff6-33c6e98102f7-kube-api-access-2x7pc\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vq928\" (UID: \"aad104bb-dc8a-48f5-9ff6-33c6e98102f7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vq928" Sep 11 20:11:00.062271 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:11:00.062193 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aad104bb-dc8a-48f5-9ff6-33c6e98102f7-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vq928\" (UID: \"aad104bb-dc8a-48f5-9ff6-33c6e98102f7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vq928" Sep 11 20:11:00.062271 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:11:00.062230 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aad104bb-dc8a-48f5-9ff6-33c6e98102f7-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vq928\" (UID: \"aad104bb-dc8a-48f5-9ff6-33c6e98102f7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vq928" Sep 11 20:11:00.062366 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:11:00.062282 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aad104bb-dc8a-48f5-9ff6-33c6e98102f7-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vq928\" (UID: \"aad104bb-dc8a-48f5-9ff6-33c6e98102f7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vq928" Sep 11 20:11:00.064113 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:11:00.064091 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aad104bb-dc8a-48f5-9ff6-33c6e98102f7-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vq928\" (UID: \"aad104bb-dc8a-48f5-9ff6-33c6e98102f7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vq928" Sep 11 20:11:00.064289 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:11:00.064272 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aad104bb-dc8a-48f5-9ff6-33c6e98102f7-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vq928\" (UID: \"aad104bb-dc8a-48f5-9ff6-33c6e98102f7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vq928" Sep 11 20:11:00.070002 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:11:00.069979 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x7pc\" (UniqueName: \"kubernetes.io/projected/aad104bb-dc8a-48f5-9ff6-33c6e98102f7-kube-api-access-2x7pc\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vq928\" (UID: \"aad104bb-dc8a-48f5-9ff6-33c6e98102f7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vq928" Sep 11 20:11:00.101667 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:11:00.101644 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vq928" Sep 11 20:11:00.372960 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:11:00.372939 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-vq928"] Sep 11 20:11:00.375016 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:11:00.374989 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaad104bb_dc8a_48f5_9ff6_33c6e98102f7.slice/crio-bf6575979ab538c23b35e5ecf9c4dae45a3db46c2b09c6b4164eb208d4e5fb58 WatchSource:0}: Error finding container bf6575979ab538c23b35e5ecf9c4dae45a3db46c2b09c6b4164eb208d4e5fb58: Status 404 returned error can't find the container with id bf6575979ab538c23b35e5ecf9c4dae45a3db46c2b09c6b4164eb208d4e5fb58 Sep 11 20:11:00.418392 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:11:00.418362 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vq928" event={"ID":"aad104bb-dc8a-48f5-9ff6-33c6e98102f7","Type":"ContainerStarted","Data":"bf6575979ab538c23b35e5ecf9c4dae45a3db46c2b09c6b4164eb208d4e5fb58"} Sep 11 20:11:01.425185 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:11:01.425148 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vq928" event={"ID":"aad104bb-dc8a-48f5-9ff6-33c6e98102f7","Type":"ContainerStarted","Data":"b34f7e979886d1d5b30784d436756c9a032183348028740eb08e0be0ceb01a7a"} Sep 11 20:11:01.446297 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:11:01.446253 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vq928" podStartSLOduration=2.236887499 podStartE2EDuration="2.446236548s" totalImagesPullingTime="209.349049ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:10:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:11:01.443769558 +0000 UTC m=+631.491819902" watchObservedRunningTime="2026-09-11 20:11:01.446236548 +0000 UTC m=+631.494286892" Sep 11 20:11:04.077679 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:11:04.077622 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffpqcl"] Sep 11 20:11:04.082113 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:11:04.082088 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffpqcl" Sep 11 20:11:04.084157 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:11:04.084133 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Sep 11 20:11:04.090374 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:11:04.090341 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffpqcl"] Sep 11 20:11:04.117201 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:11:04.117172 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:11:04.200432 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:11:04.200398 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22cd130b-ace8-4d20-a769-46fd823ace55-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffpqcl\" (UID: \"22cd130b-ace8-4d20-a769-46fd823ace55\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffpqcl" Sep 11 20:11:04.200568 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:11:04.200500 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9cmj\" (UniqueName: \"kubernetes.io/projected/22cd130b-ace8-4d20-a769-46fd823ace55-kube-api-access-n9cmj\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffpqcl\" (UID: \"22cd130b-ace8-4d20-a769-46fd823ace55\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffpqcl" Sep 11 20:11:04.200568 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:11:04.200548 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22cd130b-ace8-4d20-a769-46fd823ace55-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffpqcl\" (UID: \"22cd130b-ace8-4d20-a769-46fd823ace55\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffpqcl" Sep 11 20:11:04.200733 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:11:04.200715 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22cd130b-ace8-4d20-a769-46fd823ace55-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffpqcl\" (UID: \"22cd130b-ace8-4d20-a769-46fd823ace55\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffpqcl" Sep 11 20:11:04.200769 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:11:04.200747 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22cd130b-ace8-4d20-a769-46fd823ace55-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffpqcl\" (UID: \"22cd130b-ace8-4d20-a769-46fd823ace55\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffpqcl" Sep 11 20:11:04.200865 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:11:04.200844 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22cd130b-ace8-4d20-a769-46fd823ace55-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffpqcl\" (UID: \"22cd130b-ace8-4d20-a769-46fd823ace55\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffpqcl" Sep 11 20:11:04.302214 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:11:04.302184 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9cmj\" (UniqueName: \"kubernetes.io/projected/22cd130b-ace8-4d20-a769-46fd823ace55-kube-api-access-n9cmj\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffpqcl\" (UID: \"22cd130b-ace8-4d20-a769-46fd823ace55\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffpqcl" Sep 11 20:11:04.302355 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:11:04.302233 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22cd130b-ace8-4d20-a769-46fd823ace55-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffpqcl\" (UID: \"22cd130b-ace8-4d20-a769-46fd823ace55\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffpqcl" Sep 11 20:11:04.302355 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:11:04.302298 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22cd130b-ace8-4d20-a769-46fd823ace55-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffpqcl\" (UID: \"22cd130b-ace8-4d20-a769-46fd823ace55\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffpqcl" Sep 11 20:11:04.302355 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:11:04.302313 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22cd130b-ace8-4d20-a769-46fd823ace55-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffpqcl\" (UID: \"22cd130b-ace8-4d20-a769-46fd823ace55\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffpqcl" Sep 11 20:11:04.302355 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:11:04.302344 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22cd130b-ace8-4d20-a769-46fd823ace55-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffpqcl\" (UID: \"22cd130b-ace8-4d20-a769-46fd823ace55\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffpqcl" Sep 11 20:11:04.302609 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:11:04.302378 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22cd130b-ace8-4d20-a769-46fd823ace55-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffpqcl\" (UID: \"22cd130b-ace8-4d20-a769-46fd823ace55\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffpqcl" Sep 11 20:11:04.302843 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:11:04.302813 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22cd130b-ace8-4d20-a769-46fd823ace55-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffpqcl\" (UID: \"22cd130b-ace8-4d20-a769-46fd823ace55\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffpqcl" Sep 11 20:11:04.302843 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:11:04.302831 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22cd130b-ace8-4d20-a769-46fd823ace55-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffpqcl\" (UID: \"22cd130b-ace8-4d20-a769-46fd823ace55\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffpqcl" Sep 11 20:11:04.302987 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:11:04.302865 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22cd130b-ace8-4d20-a769-46fd823ace55-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffpqcl\" (UID: \"22cd130b-ace8-4d20-a769-46fd823ace55\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffpqcl" Sep 11 20:11:04.304639 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:11:04.304619 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22cd130b-ace8-4d20-a769-46fd823ace55-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffpqcl\" (UID: \"22cd130b-ace8-4d20-a769-46fd823ace55\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffpqcl" Sep 11 20:11:04.304820 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:11:04.304804 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22cd130b-ace8-4d20-a769-46fd823ace55-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffpqcl\" (UID: \"22cd130b-ace8-4d20-a769-46fd823ace55\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffpqcl" Sep 11 20:11:04.309291 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:11:04.309271 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9cmj\" (UniqueName: \"kubernetes.io/projected/22cd130b-ace8-4d20-a769-46fd823ace55-kube-api-access-n9cmj\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffpqcl\" (UID: \"22cd130b-ace8-4d20-a769-46fd823ace55\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffpqcl" Sep 11 20:11:04.396242 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:11:04.396212 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffpqcl" Sep 11 20:11:04.670247 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:11:04.670225 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffpqcl"] Sep 11 20:11:04.672484 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:11:04.672457 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22cd130b_ace8_4d20_a769_46fd823ace55.slice/crio-bd35c28f932b2566502ba1b0652af56ad7008108c32926dc17721039fc66a6fe WatchSource:0}: Error finding container bd35c28f932b2566502ba1b0652af56ad7008108c32926dc17721039fc66a6fe: Status 404 returned error can't find the container with id bd35c28f932b2566502ba1b0652af56ad7008108c32926dc17721039fc66a6fe Sep 11 20:11:05.447340 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:11:05.447299 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffpqcl" event={"ID":"22cd130b-ace8-4d20-a769-46fd823ace55","Type":"ContainerStarted","Data":"ccf6c59f23b8d4d749fa9a9f6e96af795bb17a34a76ba4091d74cd4a7d27cea7"} Sep 11 20:11:05.447732 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:11:05.447360 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffpqcl" event={"ID":"22cd130b-ace8-4d20-a769-46fd823ace55","Type":"ContainerStarted","Data":"bd35c28f932b2566502ba1b0652af56ad7008108c32926dc17721039fc66a6fe"} Sep 11 20:11:05.464985 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:11:05.464935 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffpqcl" podStartSLOduration=1.251577486 podStartE2EDuration="1.464919635s" totalImagesPullingTime="213.342149ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:11:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:11:05.463095775 +0000 UTC m=+635.511146145" watchObservedRunningTime="2026-09-11 20:11:05.464919635 +0000 UTC m=+635.512969978" Sep 11 20:11:10.102050 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:11:10.102012 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vq928" Sep 11 20:11:10.102472 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:11:10.102102 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vq928" Sep 11 20:11:10.114727 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:11:10.114705 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vq928" Sep 11 20:11:10.484281 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:11:10.484204 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vq928" Sep 11 20:11:14.397332 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:11:14.397289 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffpqcl" Sep 11 20:11:14.397865 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:11:14.397393 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffpqcl" Sep 11 20:11:14.409843 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:11:14.409821 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffpqcl" Sep 11 20:11:14.503437 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:11:14.503402 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffpqcl" Sep 11 20:11:24.425957 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:11:24.425918 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:11:30.528876 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:11:30.528828 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:11:59.212228 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:11:59.212195 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:11:59.312849 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:11:59.312787 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:12:38.359008 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:38.358900 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:12:38.467644 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:38.467615 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:12:39.410250 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:39.410215 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:12:39.503941 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:39.503906 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:12:54.380126 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:54.380085 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:12:54.484453 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:54.484422 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:12:54.599687 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:54.599634 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:12:54.762922 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:54.762827 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:12:54.879651 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:54.879612 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:12:54.896802 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:54.896770 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs"] Sep 11 20:12:54.900617 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:54.900595 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs" Sep 11 20:12:54.902668 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:54.902646 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-9d9abb-openshift-default-dockercfg-9cp4z\"" Sep 11 20:12:54.917692 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:54.917654 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs"] Sep 11 20:12:54.960609 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:54.960571 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:12:55.016544 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:55.016462 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7947d808-ad70-4b13-b396-4936f2588e44-workload-socket\") pod \"e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs\" (UID: \"7947d808-ad70-4b13-b396-4936f2588e44\") " pod="openshift-ingress/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs" Sep 11 20:12:55.016544 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:55.016497 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7947d808-ad70-4b13-b396-4936f2588e44-istio-podinfo\") pod \"e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs\" (UID: \"7947d808-ad70-4b13-b396-4936f2588e44\") " pod="openshift-ingress/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs" Sep 11 20:12:55.016544 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:55.016533 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7947d808-ad70-4b13-b396-4936f2588e44-credential-socket\") pod \"e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs\" (UID: \"7947d808-ad70-4b13-b396-4936f2588e44\") " pod="openshift-ingress/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs" Sep 11 20:12:55.016853 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:55.016592 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7947d808-ad70-4b13-b396-4936f2588e44-istio-token\") pod \"e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs\" (UID: \"7947d808-ad70-4b13-b396-4936f2588e44\") " pod="openshift-ingress/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs" Sep 11 20:12:55.016853 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:55.016617 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7947d808-ad70-4b13-b396-4936f2588e44-istiod-ca-cert\") pod \"e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs\" (UID: \"7947d808-ad70-4b13-b396-4936f2588e44\") " pod="openshift-ingress/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs" Sep 11 20:12:55.016853 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:55.016663 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7947d808-ad70-4b13-b396-4936f2588e44-istio-envoy\") pod \"e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs\" (UID: \"7947d808-ad70-4b13-b396-4936f2588e44\") " pod="openshift-ingress/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs" Sep 11 20:12:55.016853 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:55.016685 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7947d808-ad70-4b13-b396-4936f2588e44-workload-certs\") pod \"e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs\" (UID: \"7947d808-ad70-4b13-b396-4936f2588e44\") " pod="openshift-ingress/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs" Sep 11 20:12:55.016853 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:55.016761 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9wxd\" (UniqueName: \"kubernetes.io/projected/7947d808-ad70-4b13-b396-4936f2588e44-kube-api-access-v9wxd\") pod \"e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs\" (UID: \"7947d808-ad70-4b13-b396-4936f2588e44\") " pod="openshift-ingress/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs" Sep 11 20:12:55.016853 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:55.016803 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7947d808-ad70-4b13-b396-4936f2588e44-istio-data\") pod \"e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs\" (UID: \"7947d808-ad70-4b13-b396-4936f2588e44\") " pod="openshift-ingress/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs" Sep 11 20:12:55.117987 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:55.117949 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9wxd\" (UniqueName: \"kubernetes.io/projected/7947d808-ad70-4b13-b396-4936f2588e44-kube-api-access-v9wxd\") pod \"e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs\" (UID: \"7947d808-ad70-4b13-b396-4936f2588e44\") " pod="openshift-ingress/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs" Sep 11 20:12:55.118178 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:55.118009 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7947d808-ad70-4b13-b396-4936f2588e44-istio-data\") pod \"e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs\" (UID: \"7947d808-ad70-4b13-b396-4936f2588e44\") " pod="openshift-ingress/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs" Sep 11 20:12:55.118178 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:55.118083 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7947d808-ad70-4b13-b396-4936f2588e44-workload-socket\") pod \"e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs\" (UID: \"7947d808-ad70-4b13-b396-4936f2588e44\") " pod="openshift-ingress/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs" Sep 11 20:12:55.118178 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:55.118113 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7947d808-ad70-4b13-b396-4936f2588e44-istio-podinfo\") pod \"e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs\" (UID: \"7947d808-ad70-4b13-b396-4936f2588e44\") " pod="openshift-ingress/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs" Sep 11 20:12:55.118178 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:55.118149 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7947d808-ad70-4b13-b396-4936f2588e44-credential-socket\") pod \"e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs\" (UID: \"7947d808-ad70-4b13-b396-4936f2588e44\") " pod="openshift-ingress/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs" Sep 11 20:12:55.118402 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:55.118302 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7947d808-ad70-4b13-b396-4936f2588e44-istio-token\") pod \"e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs\" (UID: \"7947d808-ad70-4b13-b396-4936f2588e44\") " pod="openshift-ingress/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs" Sep 11 20:12:55.118402 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:55.118348 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7947d808-ad70-4b13-b396-4936f2588e44-istiod-ca-cert\") pod \"e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs\" (UID: \"7947d808-ad70-4b13-b396-4936f2588e44\") " pod="openshift-ingress/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs" Sep 11 20:12:55.118544 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:55.118401 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7947d808-ad70-4b13-b396-4936f2588e44-istio-envoy\") pod \"e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs\" (UID: \"7947d808-ad70-4b13-b396-4936f2588e44\") " pod="openshift-ingress/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs" Sep 11 20:12:55.118544 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:55.118415 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7947d808-ad70-4b13-b396-4936f2588e44-istio-data\") pod \"e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs\" (UID: \"7947d808-ad70-4b13-b396-4936f2588e44\") " pod="openshift-ingress/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs" Sep 11 20:12:55.118544 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:55.118425 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7947d808-ad70-4b13-b396-4936f2588e44-workload-socket\") pod \"e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs\" (UID: \"7947d808-ad70-4b13-b396-4936f2588e44\") " pod="openshift-ingress/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs" Sep 11 20:12:55.118544 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:55.118439 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7947d808-ad70-4b13-b396-4936f2588e44-workload-certs\") pod \"e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs\" (UID: \"7947d808-ad70-4b13-b396-4936f2588e44\") " pod="openshift-ingress/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs" Sep 11 20:12:55.118544 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:55.118521 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7947d808-ad70-4b13-b396-4936f2588e44-credential-socket\") pod \"e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs\" (UID: \"7947d808-ad70-4b13-b396-4936f2588e44\") " pod="openshift-ingress/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs" Sep 11 20:12:55.118833 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:55.118813 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7947d808-ad70-4b13-b396-4936f2588e44-workload-certs\") pod \"e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs\" (UID: \"7947d808-ad70-4b13-b396-4936f2588e44\") " pod="openshift-ingress/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs" Sep 11 20:12:55.119022 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:55.119006 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7947d808-ad70-4b13-b396-4936f2588e44-istiod-ca-cert\") pod \"e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs\" (UID: \"7947d808-ad70-4b13-b396-4936f2588e44\") " pod="openshift-ingress/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs" Sep 11 20:12:55.120473 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:55.120440 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7947d808-ad70-4b13-b396-4936f2588e44-istio-envoy\") pod \"e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs\" (UID: \"7947d808-ad70-4b13-b396-4936f2588e44\") " pod="openshift-ingress/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs" Sep 11 20:12:55.120783 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:55.120765 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7947d808-ad70-4b13-b396-4936f2588e44-istio-podinfo\") pod \"e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs\" (UID: \"7947d808-ad70-4b13-b396-4936f2588e44\") " pod="openshift-ingress/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs" Sep 11 20:12:55.128982 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:55.128958 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9wxd\" (UniqueName: \"kubernetes.io/projected/7947d808-ad70-4b13-b396-4936f2588e44-kube-api-access-v9wxd\") pod \"e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs\" (UID: \"7947d808-ad70-4b13-b396-4936f2588e44\") " pod="openshift-ingress/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs" Sep 11 20:12:55.129295 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:55.129277 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7947d808-ad70-4b13-b396-4936f2588e44-istio-token\") pod \"e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs\" (UID: \"7947d808-ad70-4b13-b396-4936f2588e44\") " pod="openshift-ingress/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs" Sep 11 20:12:55.153705 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:55.153681 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:12:55.211883 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:55.211855 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs" Sep 11 20:12:55.490386 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:55.490350 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:12:55.490725 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:55.490454 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs"] Sep 11 20:12:55.492677 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:12:55.492651 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7947d808_ad70_4b13_b396_4936f2588e44.slice/crio-3d1dfafd92868a9ce50e3e33a9af4d440d414d2119aba89590952ec2a54eba9b WatchSource:0}: Error finding container 3d1dfafd92868a9ce50e3e33a9af4d440d414d2119aba89590952ec2a54eba9b: Status 404 returned error can't find the container with id 3d1dfafd92868a9ce50e3e33a9af4d440d414d2119aba89590952ec2a54eba9b Sep 11 20:12:55.494852 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:55.494819 2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 20:12:55.494940 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:55.494885 2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 20:12:55.494940 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:55.494912 2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 20:12:55.791952 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:55.791768 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:12:55.936997 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:55.936965 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:12:55.995684 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:55.995655 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs" event={"ID":"7947d808-ad70-4b13-b396-4936f2588e44","Type":"ContainerStarted","Data":"8ae7fc1b8494c0d7e86d8740688a9e8f3391285332a2888fb1be74972b40716d"} Sep 11 20:12:55.995824 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:55.995701 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs" event={"ID":"7947d808-ad70-4b13-b396-4936f2588e44","Type":"ContainerStarted","Data":"3d1dfafd92868a9ce50e3e33a9af4d440d414d2119aba89590952ec2a54eba9b"} Sep 11 20:12:56.018538 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:56.018453 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs" podStartSLOduration=2.018434938 podStartE2EDuration="2.018434938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:12:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:12:56.017179705 +0000 UTC m=+746.065230059" watchObservedRunningTime="2026-09-11 20:12:56.018434938 +0000 UTC m=+746.066485281" Sep 11 20:12:56.212689 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:56.212656 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs" Sep 11 20:12:56.214125 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:56.214100 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:12:56.217902 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:56.217880 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs" Sep 11 20:12:57.000419 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:57.000387 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs" Sep 11 20:12:57.001419 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:57.001401 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs" Sep 11 20:12:57.620560 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:57.620528 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:12:57.790296 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:57.790259 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:12:57.894985 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:57.894909 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:12:57.995127 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:57.995097 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:12:58.093308 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:58.093259 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:12:58.961294 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:58.961244 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b3e1dd-545f764567-7qgrd"] Sep 11 20:12:58.965513 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:58.965490 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3e1dd-545f764567-7qgrd" Sep 11 20:12:58.969100 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:58.969078 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-b3e1dd-serving-cert\"" Sep 11 20:12:58.980266 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:58.980245 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b3e1dd-545f764567-7qgrd"] Sep 11 20:12:59.004224 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:59.004204 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:12:59.058192 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:59.058166 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghx5x\" (UniqueName: \"kubernetes.io/projected/42b48ad0-8671-4297-8af7-3b0cb3e6b87e-kube-api-access-ghx5x\") pod \"maas-api-e2e-inf-a-b3e1dd-545f764567-7qgrd\" (UID: \"42b48ad0-8671-4297-8af7-3b0cb3e6b87e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3e1dd-545f764567-7qgrd" Sep 11 20:12:59.058309 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:59.058204 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42b48ad0-8671-4297-8af7-3b0cb3e6b87e-metrics-certs\") pod \"maas-api-e2e-inf-a-b3e1dd-545f764567-7qgrd\" (UID: \"42b48ad0-8671-4297-8af7-3b0cb3e6b87e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3e1dd-545f764567-7qgrd" Sep 11 20:12:59.058309 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:59.058228 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42b48ad0-8671-4297-8af7-3b0cb3e6b87e-maas-api-tls\") pod \"maas-api-e2e-inf-a-b3e1dd-545f764567-7qgrd\" (UID: \"42b48ad0-8671-4297-8af7-3b0cb3e6b87e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3e1dd-545f764567-7qgrd" Sep 11 20:12:59.058309 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:59.058264 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42b48ad0-8671-4297-8af7-3b0cb3e6b87e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b3e1dd-545f764567-7qgrd\" (UID: \"42b48ad0-8671-4297-8af7-3b0cb3e6b87e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3e1dd-545f764567-7qgrd" Sep 11 20:12:59.102854 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:59.102829 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:12:59.159091 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:59.159062 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghx5x\" (UniqueName: \"kubernetes.io/projected/42b48ad0-8671-4297-8af7-3b0cb3e6b87e-kube-api-access-ghx5x\") pod \"maas-api-e2e-inf-a-b3e1dd-545f764567-7qgrd\" (UID: \"42b48ad0-8671-4297-8af7-3b0cb3e6b87e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3e1dd-545f764567-7qgrd" Sep 11 20:12:59.159216 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:59.159125 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42b48ad0-8671-4297-8af7-3b0cb3e6b87e-metrics-certs\") pod \"maas-api-e2e-inf-a-b3e1dd-545f764567-7qgrd\" (UID: \"42b48ad0-8671-4297-8af7-3b0cb3e6b87e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3e1dd-545f764567-7qgrd" Sep 11 20:12:59.159216 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:59.159157 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42b48ad0-8671-4297-8af7-3b0cb3e6b87e-maas-api-tls\") pod \"maas-api-e2e-inf-a-b3e1dd-545f764567-7qgrd\" (UID: \"42b48ad0-8671-4297-8af7-3b0cb3e6b87e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3e1dd-545f764567-7qgrd" Sep 11 20:12:59.159216 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:59.159183 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42b48ad0-8671-4297-8af7-3b0cb3e6b87e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b3e1dd-545f764567-7qgrd\" (UID: \"42b48ad0-8671-4297-8af7-3b0cb3e6b87e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3e1dd-545f764567-7qgrd" Sep 11 20:12:59.159973 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:59.159948 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42b48ad0-8671-4297-8af7-3b0cb3e6b87e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b3e1dd-545f764567-7qgrd\" (UID: \"42b48ad0-8671-4297-8af7-3b0cb3e6b87e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3e1dd-545f764567-7qgrd" Sep 11 20:12:59.161662 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:59.161639 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42b48ad0-8671-4297-8af7-3b0cb3e6b87e-metrics-certs\") pod \"maas-api-e2e-inf-a-b3e1dd-545f764567-7qgrd\" (UID: \"42b48ad0-8671-4297-8af7-3b0cb3e6b87e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3e1dd-545f764567-7qgrd" Sep 11 20:12:59.161754 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:59.161671 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42b48ad0-8671-4297-8af7-3b0cb3e6b87e-maas-api-tls\") pod \"maas-api-e2e-inf-a-b3e1dd-545f764567-7qgrd\" (UID: \"42b48ad0-8671-4297-8af7-3b0cb3e6b87e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3e1dd-545f764567-7qgrd" Sep 11 20:12:59.167998 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:59.167974 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghx5x\" (UniqueName: \"kubernetes.io/projected/42b48ad0-8671-4297-8af7-3b0cb3e6b87e-kube-api-access-ghx5x\") pod \"maas-api-e2e-inf-a-b3e1dd-545f764567-7qgrd\" (UID: \"42b48ad0-8671-4297-8af7-3b0cb3e6b87e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3e1dd-545f764567-7qgrd" Sep 11 20:12:59.207530 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:59.207495 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:12:59.279996 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:59.279932 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3e1dd-545f764567-7qgrd" Sep 11 20:12:59.564195 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:12:59.564161 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42b48ad0_8671_4297_8af7_3b0cb3e6b87e.slice/crio-ab699a15f1973ce0b89ba6c9fdad22b8d36eb336b828034e1ad5ede9d2d696f3 WatchSource:0}: Error finding container ab699a15f1973ce0b89ba6c9fdad22b8d36eb336b828034e1ad5ede9d2d696f3: Status 404 returned error can't find the container with id ab699a15f1973ce0b89ba6c9fdad22b8d36eb336b828034e1ad5ede9d2d696f3 Sep 11 20:12:59.566779 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:59.566752 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b3e1dd-545f764567-7qgrd"] Sep 11 20:12:59.734063 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:59.734018 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:12:59.833627 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:59.833531 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:12:59.847928 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:59.847897 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b3e1dd-bff5699cd-dv665"] Sep 11 20:12:59.853171 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:59.853142 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b3e1dd-bff5699cd-dv665" Sep 11 20:12:59.855424 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:59.855403 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-b3e1dd\"" Sep 11 20:12:59.865138 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:59.865108 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b3e1dd-bff5699cd-dv665"] Sep 11 20:12:59.968205 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:59.968164 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9nkw\" (UniqueName: \"kubernetes.io/projected/188761e4-5b30-40cc-85ec-f4c9b77be511-kube-api-access-q9nkw\") pod \"payload-pre-processing-e2e-inf-a-b3e1dd-bff5699cd-dv665\" (UID: \"188761e4-5b30-40cc-85ec-f4c9b77be511\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b3e1dd-bff5699cd-dv665" Sep 11 20:12:59.968369 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:12:59.968229 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/188761e4-5b30-40cc-85ec-f4c9b77be511-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b3e1dd-bff5699cd-dv665\" (UID: \"188761e4-5b30-40cc-85ec-f4c9b77be511\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b3e1dd-bff5699cd-dv665" Sep 11 20:13:00.018215 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:00.018182 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3e1dd-545f764567-7qgrd" event={"ID":"42b48ad0-8671-4297-8af7-3b0cb3e6b87e","Type":"ContainerStarted","Data":"ab699a15f1973ce0b89ba6c9fdad22b8d36eb336b828034e1ad5ede9d2d696f3"} Sep 11 20:13:00.069621 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:00.069217 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9nkw\" (UniqueName: \"kubernetes.io/projected/188761e4-5b30-40cc-85ec-f4c9b77be511-kube-api-access-q9nkw\") pod \"payload-pre-processing-e2e-inf-a-b3e1dd-bff5699cd-dv665\" (UID: \"188761e4-5b30-40cc-85ec-f4c9b77be511\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b3e1dd-bff5699cd-dv665" Sep 11 20:13:00.069621 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:00.069283 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/188761e4-5b30-40cc-85ec-f4c9b77be511-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b3e1dd-bff5699cd-dv665\" (UID: \"188761e4-5b30-40cc-85ec-f4c9b77be511\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b3e1dd-bff5699cd-dv665" Sep 11 20:13:00.070263 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:00.070239 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/188761e4-5b30-40cc-85ec-f4c9b77be511-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b3e1dd-bff5699cd-dv665\" (UID: \"188761e4-5b30-40cc-85ec-f4c9b77be511\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b3e1dd-bff5699cd-dv665" Sep 11 20:13:00.080974 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:00.080950 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9nkw\" (UniqueName: \"kubernetes.io/projected/188761e4-5b30-40cc-85ec-f4c9b77be511-kube-api-access-q9nkw\") pod \"payload-pre-processing-e2e-inf-a-b3e1dd-bff5699cd-dv665\" (UID: \"188761e4-5b30-40cc-85ec-f4c9b77be511\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b3e1dd-bff5699cd-dv665" Sep 11 20:13:00.166840 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:00.166800 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b3e1dd-bff5699cd-dv665" Sep 11 20:13:00.198454 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:00.198414 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-22fc84-7fbff7d55d-zhngj"] Sep 11 20:13:00.203105 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:00.203080 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-22fc84-7fbff7d55d-zhngj" Sep 11 20:13:00.205226 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:00.205200 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-22fc84-serving-cert\"" Sep 11 20:13:00.216702 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:00.215699 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-22fc84-7fbff7d55d-zhngj"] Sep 11 20:13:00.222698 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:00.222668 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:00.294292 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:00.294261 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:00.372023 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:00.371992 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2647260f-49fc-450a-9e4b-fda58dfffddc-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-22fc84-7fbff7d55d-zhngj\" (UID: \"2647260f-49fc-450a-9e4b-fda58dfffddc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-22fc84-7fbff7d55d-zhngj" Sep 11 20:13:00.372177 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:00.372038 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjw5t\" (UniqueName: \"kubernetes.io/projected/2647260f-49fc-450a-9e4b-fda58dfffddc-kube-api-access-hjw5t\") pod \"maas-api-e2e-worker-w4-22fc84-7fbff7d55d-zhngj\" (UID: \"2647260f-49fc-450a-9e4b-fda58dfffddc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-22fc84-7fbff7d55d-zhngj" Sep 11 20:13:00.372177 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:00.372082 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2647260f-49fc-450a-9e4b-fda58dfffddc-metrics-certs\") pod \"maas-api-e2e-worker-w4-22fc84-7fbff7d55d-zhngj\" (UID: \"2647260f-49fc-450a-9e4b-fda58dfffddc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-22fc84-7fbff7d55d-zhngj" Sep 11 20:13:00.372177 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:00.372156 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2647260f-49fc-450a-9e4b-fda58dfffddc-maas-api-tls\") pod \"maas-api-e2e-worker-w4-22fc84-7fbff7d55d-zhngj\" (UID: \"2647260f-49fc-450a-9e4b-fda58dfffddc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-22fc84-7fbff7d55d-zhngj" Sep 11 20:13:00.461115 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:00.461090 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b3e1dd-bff5699cd-dv665"] Sep 11 20:13:00.463519 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:13:00.463494 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod188761e4_5b30_40cc_85ec_f4c9b77be511.slice/crio-faf67a33158fb654c77e963f93b99d74298ec0ef0f705bde5e1ac42e35b5592c WatchSource:0}: Error finding container faf67a33158fb654c77e963f93b99d74298ec0ef0f705bde5e1ac42e35b5592c: Status 404 returned error can't find the container with id faf67a33158fb654c77e963f93b99d74298ec0ef0f705bde5e1ac42e35b5592c Sep 11 20:13:00.473182 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:00.473158 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjw5t\" (UniqueName: \"kubernetes.io/projected/2647260f-49fc-450a-9e4b-fda58dfffddc-kube-api-access-hjw5t\") pod \"maas-api-e2e-worker-w4-22fc84-7fbff7d55d-zhngj\" (UID: \"2647260f-49fc-450a-9e4b-fda58dfffddc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-22fc84-7fbff7d55d-zhngj" Sep 11 20:13:00.473275 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:00.473205 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2647260f-49fc-450a-9e4b-fda58dfffddc-metrics-certs\") pod \"maas-api-e2e-worker-w4-22fc84-7fbff7d55d-zhngj\" (UID: \"2647260f-49fc-450a-9e4b-fda58dfffddc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-22fc84-7fbff7d55d-zhngj" Sep 11 20:13:00.473275 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:00.473256 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2647260f-49fc-450a-9e4b-fda58dfffddc-maas-api-tls\") pod \"maas-api-e2e-worker-w4-22fc84-7fbff7d55d-zhngj\" (UID: \"2647260f-49fc-450a-9e4b-fda58dfffddc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-22fc84-7fbff7d55d-zhngj" Sep 11 20:13:00.473402 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:00.473333 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2647260f-49fc-450a-9e4b-fda58dfffddc-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-22fc84-7fbff7d55d-zhngj\" (UID: \"2647260f-49fc-450a-9e4b-fda58dfffddc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-22fc84-7fbff7d55d-zhngj" Sep 11 20:13:00.474078 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:00.474051 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2647260f-49fc-450a-9e4b-fda58dfffddc-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-22fc84-7fbff7d55d-zhngj\" (UID: \"2647260f-49fc-450a-9e4b-fda58dfffddc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-22fc84-7fbff7d55d-zhngj" Sep 11 20:13:00.475780 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:00.475755 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2647260f-49fc-450a-9e4b-fda58dfffddc-maas-api-tls\") pod \"maas-api-e2e-worker-w4-22fc84-7fbff7d55d-zhngj\" (UID: \"2647260f-49fc-450a-9e4b-fda58dfffddc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-22fc84-7fbff7d55d-zhngj" Sep 11 20:13:00.476027 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:00.476006 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2647260f-49fc-450a-9e4b-fda58dfffddc-metrics-certs\") pod \"maas-api-e2e-worker-w4-22fc84-7fbff7d55d-zhngj\" (UID: \"2647260f-49fc-450a-9e4b-fda58dfffddc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-22fc84-7fbff7d55d-zhngj" Sep 11 20:13:00.482957 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:00.482930 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjw5t\" (UniqueName: \"kubernetes.io/projected/2647260f-49fc-450a-9e4b-fda58dfffddc-kube-api-access-hjw5t\") pod \"maas-api-e2e-worker-w4-22fc84-7fbff7d55d-zhngj\" (UID: \"2647260f-49fc-450a-9e4b-fda58dfffddc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-22fc84-7fbff7d55d-zhngj" Sep 11 20:13:00.501361 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:00.501319 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:00.519550 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:00.519526 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-22fc84-7fbff7d55d-zhngj" Sep 11 20:13:00.744373 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:00.744286 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:00.830371 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:00.830344 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-22fc84-7fbff7d55d-zhngj"] Sep 11 20:13:00.831878 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:13:00.831847 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2647260f_49fc_450a_9e4b_fda58dfffddc.slice/crio-0bcd166e80b3719b155209e9217cc681ee1fb8d4b7d27cc553a21d6f319281b6 WatchSource:0}: Error finding container 0bcd166e80b3719b155209e9217cc681ee1fb8d4b7d27cc553a21d6f319281b6: Status 404 returned error can't find the container with id 0bcd166e80b3719b155209e9217cc681ee1fb8d4b7d27cc553a21d6f319281b6 Sep 11 20:13:00.869882 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:00.869857 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:01.025258 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:01.025158 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-22fc84-7fbff7d55d-zhngj" event={"ID":"2647260f-49fc-450a-9e4b-fda58dfffddc","Type":"ContainerStarted","Data":"0bcd166e80b3719b155209e9217cc681ee1fb8d4b7d27cc553a21d6f319281b6"} Sep 11 20:13:01.027301 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:01.027264 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3e1dd-545f764567-7qgrd" event={"ID":"42b48ad0-8671-4297-8af7-3b0cb3e6b87e","Type":"ContainerStarted","Data":"cecd0a66fc9565732da59084cbbeb41055e2b3962582acacaf4a68eec1aad5b9"} Sep 11 20:13:01.027554 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:01.027342 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3e1dd-545f764567-7qgrd" Sep 11 20:13:01.029205 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:01.029176 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b3e1dd-bff5699cd-dv665" event={"ID":"188761e4-5b30-40cc-85ec-f4c9b77be511","Type":"ContainerStarted","Data":"8729ea5d0fc95ba31d3b171ac9dd28b6da203c8d16c57e792ea35c613559e32e"} Sep 11 20:13:01.029310 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:01.029219 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b3e1dd-bff5699cd-dv665" event={"ID":"188761e4-5b30-40cc-85ec-f4c9b77be511","Type":"ContainerStarted","Data":"faf67a33158fb654c77e963f93b99d74298ec0ef0f705bde5e1ac42e35b5592c"} Sep 11 20:13:01.029310 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:01.029249 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b3e1dd-bff5699cd-dv665" Sep 11 20:13:01.053187 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:01.053138 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3e1dd-545f764567-7qgrd" podStartSLOduration=2.632977012 podStartE2EDuration="3.053125009s" totalImagesPullingTime="420.147997ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:12:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:13:01.051274176 +0000 UTC m=+751.099324520" watchObservedRunningTime="2026-09-11 20:13:01.053125009 +0000 UTC m=+751.101175362" Sep 11 20:13:01.078085 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:01.078043 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b3e1dd-bff5699cd-dv665" podStartSLOduration=2.078028016 podStartE2EDuration="2.078028016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:12:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:13:01.07683852 +0000 UTC m=+751.124888875" watchObservedRunningTime="2026-09-11 20:13:01.078028016 +0000 UTC m=+751.126078391" Sep 11 20:13:01.227699 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:01.225074 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-9d9abb-564996c49d-2x6rg"] Sep 11 20:13:01.230609 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:01.230120 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9d9abb-564996c49d-2x6rg" Sep 11 20:13:01.232641 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:01.232453 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-9d9abb-serving-cert\"" Sep 11 20:13:01.239982 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:01.239926 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-9d9abb-564996c49d-2x6rg"] Sep 11 20:13:01.298360 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:01.298334 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:01.382524 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:01.382481 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26126c9a-d580-4890-909b-ad74419c775d-metrics-certs\") pod \"maas-api-e2e-worker-w0-9d9abb-564996c49d-2x6rg\" (UID: \"26126c9a-d580-4890-909b-ad74419c775d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9d9abb-564996c49d-2x6rg" Sep 11 20:13:01.382859 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:01.382831 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26126c9a-d580-4890-909b-ad74419c775d-maas-api-tls\") pod \"maas-api-e2e-worker-w0-9d9abb-564996c49d-2x6rg\" (UID: \"26126c9a-d580-4890-909b-ad74419c775d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9d9abb-564996c49d-2x6rg" Sep 11 20:13:01.383016 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:01.383000 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26126c9a-d580-4890-909b-ad74419c775d-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-9d9abb-564996c49d-2x6rg\" (UID: \"26126c9a-d580-4890-909b-ad74419c775d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9d9abb-564996c49d-2x6rg" Sep 11 20:13:01.383207 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:01.383170 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6snvp\" (UniqueName: \"kubernetes.io/projected/26126c9a-d580-4890-909b-ad74419c775d-kube-api-access-6snvp\") pod \"maas-api-e2e-worker-w0-9d9abb-564996c49d-2x6rg\" (UID: \"26126c9a-d580-4890-909b-ad74419c775d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9d9abb-564996c49d-2x6rg" Sep 11 20:13:01.401627 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:01.401605 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:01.484121 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:01.484086 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26126c9a-d580-4890-909b-ad74419c775d-metrics-certs\") pod \"maas-api-e2e-worker-w0-9d9abb-564996c49d-2x6rg\" (UID: \"26126c9a-d580-4890-909b-ad74419c775d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9d9abb-564996c49d-2x6rg" Sep 11 20:13:01.484288 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:01.484120 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26126c9a-d580-4890-909b-ad74419c775d-maas-api-tls\") pod \"maas-api-e2e-worker-w0-9d9abb-564996c49d-2x6rg\" (UID: \"26126c9a-d580-4890-909b-ad74419c775d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9d9abb-564996c49d-2x6rg" Sep 11 20:13:01.484288 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:01.484182 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26126c9a-d580-4890-909b-ad74419c775d-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-9d9abb-564996c49d-2x6rg\" (UID: \"26126c9a-d580-4890-909b-ad74419c775d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9d9abb-564996c49d-2x6rg" Sep 11 20:13:01.484288 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:01.484258 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6snvp\" (UniqueName: \"kubernetes.io/projected/26126c9a-d580-4890-909b-ad74419c775d-kube-api-access-6snvp\") pod \"maas-api-e2e-worker-w0-9d9abb-564996c49d-2x6rg\" (UID: \"26126c9a-d580-4890-909b-ad74419c775d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9d9abb-564996c49d-2x6rg" Sep 11 20:13:01.485009 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:01.484983 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26126c9a-d580-4890-909b-ad74419c775d-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-9d9abb-564996c49d-2x6rg\" (UID: \"26126c9a-d580-4890-909b-ad74419c775d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9d9abb-564996c49d-2x6rg" Sep 11 20:13:01.487204 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:01.487181 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26126c9a-d580-4890-909b-ad74419c775d-maas-api-tls\") pod \"maas-api-e2e-worker-w0-9d9abb-564996c49d-2x6rg\" (UID: \"26126c9a-d580-4890-909b-ad74419c775d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9d9abb-564996c49d-2x6rg" Sep 11 20:13:01.487307 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:01.487215 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26126c9a-d580-4890-909b-ad74419c775d-metrics-certs\") pod \"maas-api-e2e-worker-w0-9d9abb-564996c49d-2x6rg\" (UID: \"26126c9a-d580-4890-909b-ad74419c775d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9d9abb-564996c49d-2x6rg" Sep 11 20:13:01.492823 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:01.492800 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6snvp\" (UniqueName: \"kubernetes.io/projected/26126c9a-d580-4890-909b-ad74419c775d-kube-api-access-6snvp\") pod \"maas-api-e2e-worker-w0-9d9abb-564996c49d-2x6rg\" (UID: \"26126c9a-d580-4890-909b-ad74419c775d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9d9abb-564996c49d-2x6rg" Sep 11 20:13:01.565773 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:01.565714 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9d9abb-564996c49d-2x6rg" Sep 11 20:13:01.846596 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:01.846287 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-9d9abb-564996c49d-2x6rg"] Sep 11 20:13:01.854111 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:13:01.854010 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26126c9a_d580_4890_909b_ad74419c775d.slice/crio-94f8bdd478481cd3bdcdd69522314c7339691be70c43bb28d6332031b31a5d70 WatchSource:0}: Error finding container 94f8bdd478481cd3bdcdd69522314c7339691be70c43bb28d6332031b31a5d70: Status 404 returned error can't find the container with id 94f8bdd478481cd3bdcdd69522314c7339691be70c43bb28d6332031b31a5d70 Sep 11 20:13:01.898942 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:01.898906 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:01.983213 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:01.983176 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:02.036730 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:02.036694 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9d9abb-564996c49d-2x6rg" event={"ID":"26126c9a-d580-4890-909b-ad74419c775d","Type":"ContainerStarted","Data":"94f8bdd478481cd3bdcdd69522314c7339691be70c43bb28d6332031b31a5d70"} Sep 11 20:13:02.038475 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:02.038425 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-22fc84-7fbff7d55d-zhngj" event={"ID":"2647260f-49fc-450a-9e4b-fda58dfffddc","Type":"ContainerStarted","Data":"14ffec2bce9d656fcc9d3be80234e311428fa15c559502bd54799d14735fd252"} Sep 11 20:13:02.061286 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:02.061240 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-22fc84-7fbff7d55d-zhngj" podStartSLOduration=1.67866062 podStartE2EDuration="2.061225313s" totalImagesPullingTime="382.564693ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:13:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:13:02.059742644 +0000 UTC m=+752.107792988" watchObservedRunningTime="2026-09-11 20:13:02.061225313 +0000 UTC m=+752.109275657" Sep 11 20:13:02.434301 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:02.434264 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:03.045673 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:03.045632 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9d9abb-564996c49d-2x6rg" event={"ID":"26126c9a-d580-4890-909b-ad74419c775d","Type":"ContainerStarted","Data":"d4439ce5346d8b22ee60cb98a9ff790cf68fef4d1db536a779b291a09c26992e"} Sep 11 20:13:03.045829 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:03.045757 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-22fc84-7fbff7d55d-zhngj" Sep 11 20:13:03.074032 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:03.073990 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9d9abb-564996c49d-2x6rg" podStartSLOduration=1.743648203 podStartE2EDuration="2.073975367s" totalImagesPullingTime="330.327164ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:13:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:13:03.072016651 +0000 UTC m=+753.120066996" watchObservedRunningTime="2026-09-11 20:13:03.073975367 +0000 UTC m=+753.122025710" Sep 11 20:13:03.186568 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:03.186539 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:03.354046 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:03.353992 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:03.476991 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:03.476953 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:03.573510 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:03.573476 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:03.673808 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:03.673726 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:03.778449 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:03.778416 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:03.892560 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:03.892519 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:04.051492 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:04.051406 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:04.052453 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:04.052430 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9d9abb-564996c49d-2x6rg" Sep 11 20:13:04.183767 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:04.183733 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:05.530827 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:05.530793 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:06.215051 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:06.215013 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:06.391381 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:06.391347 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:06.768174 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:06.768139 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:06.863023 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:06.862983 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:07.044507 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:07.044428 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3e1dd-545f764567-7qgrd" Sep 11 20:13:07.933496 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:07.933462 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:09.026838 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:09.026794 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:09.058351 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:09.058316 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-22fc84-7fbff7d55d-zhngj" Sep 11 20:13:09.800825 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:09.800791 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-46da77-8b9b66cc9-zzk5q"] Sep 11 20:13:09.805647 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:09.805624 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-46da77-8b9b66cc9-zzk5q" Sep 11 20:13:09.808551 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:09.808530 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-46da77-serving-cert\"" Sep 11 20:13:09.816858 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:09.816829 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-46da77-8b9b66cc9-zzk5q"] Sep 11 20:13:09.856118 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:09.856090 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:09.956965 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:09.956932 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:09.970336 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:09.970309 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed23046d-f478-42bd-a143-f0add41d8342-maas-api-tls\") pod \"maas-api-e2e-autores-46da77-8b9b66cc9-zzk5q\" (UID: \"ed23046d-f478-42bd-a143-f0add41d8342\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-46da77-8b9b66cc9-zzk5q" Sep 11 20:13:09.970494 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:09.970418 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8tw7\" (UniqueName: \"kubernetes.io/projected/ed23046d-f478-42bd-a143-f0add41d8342-kube-api-access-n8tw7\") pod \"maas-api-e2e-autores-46da77-8b9b66cc9-zzk5q\" (UID: \"ed23046d-f478-42bd-a143-f0add41d8342\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-46da77-8b9b66cc9-zzk5q" Sep 11 20:13:09.970494 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:09.970457 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed23046d-f478-42bd-a143-f0add41d8342-openshift-service-ca\") pod \"maas-api-e2e-autores-46da77-8b9b66cc9-zzk5q\" (UID: \"ed23046d-f478-42bd-a143-f0add41d8342\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-46da77-8b9b66cc9-zzk5q" Sep 11 20:13:09.970494 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:09.970484 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed23046d-f478-42bd-a143-f0add41d8342-metrics-certs\") pod \"maas-api-e2e-autores-46da77-8b9b66cc9-zzk5q\" (UID: \"ed23046d-f478-42bd-a143-f0add41d8342\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-46da77-8b9b66cc9-zzk5q" Sep 11 20:13:10.063569 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:10.063493 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9d9abb-564996c49d-2x6rg" Sep 11 20:13:10.071480 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:10.071451 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8tw7\" (UniqueName: \"kubernetes.io/projected/ed23046d-f478-42bd-a143-f0add41d8342-kube-api-access-n8tw7\") pod \"maas-api-e2e-autores-46da77-8b9b66cc9-zzk5q\" (UID: \"ed23046d-f478-42bd-a143-f0add41d8342\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-46da77-8b9b66cc9-zzk5q" Sep 11 20:13:10.071626 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:10.071487 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed23046d-f478-42bd-a143-f0add41d8342-openshift-service-ca\") pod \"maas-api-e2e-autores-46da77-8b9b66cc9-zzk5q\" (UID: \"ed23046d-f478-42bd-a143-f0add41d8342\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-46da77-8b9b66cc9-zzk5q" Sep 11 20:13:10.071626 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:10.071509 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed23046d-f478-42bd-a143-f0add41d8342-metrics-certs\") pod \"maas-api-e2e-autores-46da77-8b9b66cc9-zzk5q\" (UID: \"ed23046d-f478-42bd-a143-f0add41d8342\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-46da77-8b9b66cc9-zzk5q" Sep 11 20:13:10.071626 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:10.071604 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed23046d-f478-42bd-a143-f0add41d8342-maas-api-tls\") pod \"maas-api-e2e-autores-46da77-8b9b66cc9-zzk5q\" (UID: \"ed23046d-f478-42bd-a143-f0add41d8342\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-46da77-8b9b66cc9-zzk5q" Sep 11 20:13:10.072214 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:10.072193 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed23046d-f478-42bd-a143-f0add41d8342-openshift-service-ca\") pod \"maas-api-e2e-autores-46da77-8b9b66cc9-zzk5q\" (UID: \"ed23046d-f478-42bd-a143-f0add41d8342\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-46da77-8b9b66cc9-zzk5q" Sep 11 20:13:10.074006 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:10.073985 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed23046d-f478-42bd-a143-f0add41d8342-maas-api-tls\") pod \"maas-api-e2e-autores-46da77-8b9b66cc9-zzk5q\" (UID: \"ed23046d-f478-42bd-a143-f0add41d8342\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-46da77-8b9b66cc9-zzk5q" Sep 11 20:13:10.074076 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:10.074032 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed23046d-f478-42bd-a143-f0add41d8342-metrics-certs\") pod \"maas-api-e2e-autores-46da77-8b9b66cc9-zzk5q\" (UID: \"ed23046d-f478-42bd-a143-f0add41d8342\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-46da77-8b9b66cc9-zzk5q" Sep 11 20:13:10.078772 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:10.078749 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8tw7\" (UniqueName: \"kubernetes.io/projected/ed23046d-f478-42bd-a143-f0add41d8342-kube-api-access-n8tw7\") pod \"maas-api-e2e-autores-46da77-8b9b66cc9-zzk5q\" (UID: \"ed23046d-f478-42bd-a143-f0add41d8342\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-46da77-8b9b66cc9-zzk5q" Sep 11 20:13:10.119685 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:10.119650 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-46da77-8b9b66cc9-zzk5q" Sep 11 20:13:10.369827 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:10.369794 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:10.445601 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:10.443818 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-46da77-8b9b66cc9-zzk5q"] Sep 11 20:13:10.469515 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:10.469493 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:10.978946 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:10.978910 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8e5fddd7-67d6795d64-d4cwk"] Sep 11 20:13:10.983388 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:10.983364 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e5fddd7-67d6795d64-d4cwk" Sep 11 20:13:10.985593 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:10.985557 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-8e5fddd7-serving-cert\"" Sep 11 20:13:10.993304 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:10.993267 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8e5fddd7-67d6795d64-d4cwk"] Sep 11 20:13:11.022027 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:11.021958 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:11.083224 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:11.083190 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c85c1bd1-4890-4653-ab61-cda78116644d-maas-api-tls\") pod \"maas-api-e2e-ait-8e5fddd7-67d6795d64-d4cwk\" (UID: \"c85c1bd1-4890-4653-ab61-cda78116644d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e5fddd7-67d6795d64-d4cwk" Sep 11 20:13:11.083611 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:11.083242 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c85c1bd1-4890-4653-ab61-cda78116644d-metrics-certs\") pod \"maas-api-e2e-ait-8e5fddd7-67d6795d64-d4cwk\" (UID: \"c85c1bd1-4890-4653-ab61-cda78116644d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e5fddd7-67d6795d64-d4cwk" Sep 11 20:13:11.083611 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:11.083276 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9fnm\" (UniqueName: \"kubernetes.io/projected/c85c1bd1-4890-4653-ab61-cda78116644d-kube-api-access-l9fnm\") pod \"maas-api-e2e-ait-8e5fddd7-67d6795d64-d4cwk\" (UID: \"c85c1bd1-4890-4653-ab61-cda78116644d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e5fddd7-67d6795d64-d4cwk" Sep 11 20:13:11.083611 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:11.083346 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c85c1bd1-4890-4653-ab61-cda78116644d-openshift-service-ca\") pod \"maas-api-e2e-ait-8e5fddd7-67d6795d64-d4cwk\" (UID: \"c85c1bd1-4890-4653-ab61-cda78116644d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e5fddd7-67d6795d64-d4cwk" Sep 11 20:13:11.092463 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:11.092436 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-46da77-8b9b66cc9-zzk5q" event={"ID":"ed23046d-f478-42bd-a143-f0add41d8342","Type":"ContainerStarted","Data":"9fc453054bb0c07b7f4dab076db7a249d3c4926b359ab19f8dba5a0aa2b55d3c"} Sep 11 20:13:11.092635 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:11.092483 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-46da77-8b9b66cc9-zzk5q" event={"ID":"ed23046d-f478-42bd-a143-f0add41d8342","Type":"ContainerStarted","Data":"33286e6cc03341fd9d9e2ac5e209713f66816aaa21857641de1b2bf775423c83"} Sep 11 20:13:11.092635 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:11.092524 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-46da77-8b9b66cc9-zzk5q" Sep 11 20:13:11.131607 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:11.131550 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:11.184641 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:11.184601 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9fnm\" (UniqueName: \"kubernetes.io/projected/c85c1bd1-4890-4653-ab61-cda78116644d-kube-api-access-l9fnm\") pod \"maas-api-e2e-ait-8e5fddd7-67d6795d64-d4cwk\" (UID: \"c85c1bd1-4890-4653-ab61-cda78116644d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e5fddd7-67d6795d64-d4cwk" Sep 11 20:13:11.184810 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:11.184714 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c85c1bd1-4890-4653-ab61-cda78116644d-openshift-service-ca\") pod \"maas-api-e2e-ait-8e5fddd7-67d6795d64-d4cwk\" (UID: \"c85c1bd1-4890-4653-ab61-cda78116644d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e5fddd7-67d6795d64-d4cwk" Sep 11 20:13:11.184810 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:11.184752 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c85c1bd1-4890-4653-ab61-cda78116644d-maas-api-tls\") pod \"maas-api-e2e-ait-8e5fddd7-67d6795d64-d4cwk\" (UID: \"c85c1bd1-4890-4653-ab61-cda78116644d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e5fddd7-67d6795d64-d4cwk" Sep 11 20:13:11.184810 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:11.184802 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c85c1bd1-4890-4653-ab61-cda78116644d-metrics-certs\") pod \"maas-api-e2e-ait-8e5fddd7-67d6795d64-d4cwk\" (UID: \"c85c1bd1-4890-4653-ab61-cda78116644d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e5fddd7-67d6795d64-d4cwk" Sep 11 20:13:11.185487 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:11.185463 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c85c1bd1-4890-4653-ab61-cda78116644d-openshift-service-ca\") pod \"maas-api-e2e-ait-8e5fddd7-67d6795d64-d4cwk\" (UID: \"c85c1bd1-4890-4653-ab61-cda78116644d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e5fddd7-67d6795d64-d4cwk" Sep 11 20:13:11.187215 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:11.187193 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c85c1bd1-4890-4653-ab61-cda78116644d-maas-api-tls\") pod \"maas-api-e2e-ait-8e5fddd7-67d6795d64-d4cwk\" (UID: \"c85c1bd1-4890-4653-ab61-cda78116644d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e5fddd7-67d6795d64-d4cwk" Sep 11 20:13:11.187312 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:11.187252 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c85c1bd1-4890-4653-ab61-cda78116644d-metrics-certs\") pod \"maas-api-e2e-ait-8e5fddd7-67d6795d64-d4cwk\" (UID: \"c85c1bd1-4890-4653-ab61-cda78116644d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e5fddd7-67d6795d64-d4cwk" Sep 11 20:13:11.215906 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:11.215882 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9fnm\" (UniqueName: \"kubernetes.io/projected/c85c1bd1-4890-4653-ab61-cda78116644d-kube-api-access-l9fnm\") pod \"maas-api-e2e-ait-8e5fddd7-67d6795d64-d4cwk\" (UID: \"c85c1bd1-4890-4653-ab61-cda78116644d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e5fddd7-67d6795d64-d4cwk" Sep 11 20:13:11.294366 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:11.294250 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:11.297404 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:11.297376 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e5fddd7-67d6795d64-d4cwk" Sep 11 20:13:11.596234 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:11.596101 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-46da77-8b9b66cc9-zzk5q" podStartSLOduration=2.282000543 podStartE2EDuration="2.596079679s" totalImagesPullingTime="314.079136ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:13:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:13:11.149427108 +0000 UTC m=+761.197477453" watchObservedRunningTime="2026-09-11 20:13:11.596079679 +0000 UTC m=+761.644130024" Sep 11 20:13:11.598160 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:11.598133 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8e5fddd7-67d6795d64-d4cwk"] Sep 11 20:13:11.599897 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:13:11.599837 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc85c1bd1_4890_4653_ab61_cda78116644d.slice/crio-814d25898082ffafd2ffbaa65dcc95cf9a33d93bdb3af9cfa6333f508d41b0ca WatchSource:0}: Error finding container 814d25898082ffafd2ffbaa65dcc95cf9a33d93bdb3af9cfa6333f508d41b0ca: Status 404 returned error can't find the container with id 814d25898082ffafd2ffbaa65dcc95cf9a33d93bdb3af9cfa6333f508d41b0ca Sep 11 20:13:11.616338 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:11.616308 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:11.715614 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:11.715561 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:11.795536 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:11.795503 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8e5fddd7-846789f8cd-pbccj"] Sep 11 20:13:11.800628 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:11.800613 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8e5fddd7-846789f8cd-pbccj" Sep 11 20:13:11.802739 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:11.802717 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-8e5fddd7-dockercfg-4rdgt\"" Sep 11 20:13:11.802851 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:11.802751 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-8e5fddd7\"" Sep 11 20:13:11.807942 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:11.807867 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8e5fddd7-846789f8cd-pbccj"] Sep 11 20:13:11.890138 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:11.890114 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7xz2\" (UniqueName: \"kubernetes.io/projected/579d1e65-a172-4b48-b328-1eece36b3a08-kube-api-access-v7xz2\") pod \"payload-pre-processing-e2e-ait-8e5fddd7-846789f8cd-pbccj\" (UID: \"579d1e65-a172-4b48-b328-1eece36b3a08\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8e5fddd7-846789f8cd-pbccj" Sep 11 20:13:11.890226 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:11.890165 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/579d1e65-a172-4b48-b328-1eece36b3a08-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8e5fddd7-846789f8cd-pbccj\" (UID: \"579d1e65-a172-4b48-b328-1eece36b3a08\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8e5fddd7-846789f8cd-pbccj" Sep 11 20:13:11.990839 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:11.990796 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/579d1e65-a172-4b48-b328-1eece36b3a08-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8e5fddd7-846789f8cd-pbccj\" (UID: \"579d1e65-a172-4b48-b328-1eece36b3a08\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8e5fddd7-846789f8cd-pbccj" Sep 11 20:13:11.990985 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:11.990972 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7xz2\" (UniqueName: \"kubernetes.io/projected/579d1e65-a172-4b48-b328-1eece36b3a08-kube-api-access-v7xz2\") pod \"payload-pre-processing-e2e-ait-8e5fddd7-846789f8cd-pbccj\" (UID: \"579d1e65-a172-4b48-b328-1eece36b3a08\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8e5fddd7-846789f8cd-pbccj" Sep 11 20:13:11.991707 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:11.991680 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/579d1e65-a172-4b48-b328-1eece36b3a08-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8e5fddd7-846789f8cd-pbccj\" (UID: \"579d1e65-a172-4b48-b328-1eece36b3a08\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8e5fddd7-846789f8cd-pbccj" Sep 11 20:13:12.002161 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:12.002130 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7xz2\" (UniqueName: \"kubernetes.io/projected/579d1e65-a172-4b48-b328-1eece36b3a08-kube-api-access-v7xz2\") pod \"payload-pre-processing-e2e-ait-8e5fddd7-846789f8cd-pbccj\" (UID: \"579d1e65-a172-4b48-b328-1eece36b3a08\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8e5fddd7-846789f8cd-pbccj" Sep 11 20:13:12.047611 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:12.046723 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b3e1dd-bff5699cd-dv665" Sep 11 20:13:12.099423 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:12.099384 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e5fddd7-67d6795d64-d4cwk" event={"ID":"c85c1bd1-4890-4653-ab61-cda78116644d","Type":"ContainerStarted","Data":"bb76d27c90d069cdadaf1c03a4db7fbadff5f55aa44fbea005944eab24c7f625"} Sep 11 20:13:12.099858 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:12.099440 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e5fddd7-67d6795d64-d4cwk" event={"ID":"c85c1bd1-4890-4653-ab61-cda78116644d","Type":"ContainerStarted","Data":"814d25898082ffafd2ffbaa65dcc95cf9a33d93bdb3af9cfa6333f508d41b0ca"} Sep 11 20:13:12.099858 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:12.099603 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e5fddd7-67d6795d64-d4cwk" Sep 11 20:13:12.115694 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:12.115550 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8e5fddd7-846789f8cd-pbccj" Sep 11 20:13:12.118537 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:12.118043 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e5fddd7-67d6795d64-d4cwk" podStartSLOduration=1.834472518 podStartE2EDuration="2.118025092s" totalImagesPullingTime="283.552574ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:13:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:13:12.116502557 +0000 UTC m=+762.164552901" watchObservedRunningTime="2026-09-11 20:13:12.118025092 +0000 UTC m=+762.166075439" Sep 11 20:13:12.256914 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:12.256875 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:12.415655 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:12.415630 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8e5fddd7-846789f8cd-pbccj"] Sep 11 20:13:12.417877 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:13:12.417851 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod579d1e65_a172_4b48_b328_1eece36b3a08.slice/crio-1edae705445fe46b60f596ec2da34de38f1585faff1a161255f0425ef0735b64 WatchSource:0}: Error finding container 1edae705445fe46b60f596ec2da34de38f1585faff1a161255f0425ef0735b64: Status 404 returned error can't find the container with id 1edae705445fe46b60f596ec2da34de38f1585faff1a161255f0425ef0735b64 Sep 11 20:13:12.472717 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:12.472691 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:12.794336 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:12.794253 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:13.106537 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:13.106506 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8e5fddd7-846789f8cd-pbccj" event={"ID":"579d1e65-a172-4b48-b328-1eece36b3a08","Type":"ContainerStarted","Data":"47be490458af7a17a4b5b2a82e1ec9a3cbcf7022d74e7e7bae08dd71ffb36ed6"} Sep 11 20:13:13.106971 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:13.106945 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8e5fddd7-846789f8cd-pbccj" event={"ID":"579d1e65-a172-4b48-b328-1eece36b3a08","Type":"ContainerStarted","Data":"1edae705445fe46b60f596ec2da34de38f1585faff1a161255f0425ef0735b64"} Sep 11 20:13:13.107083 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:13.106982 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8e5fddd7-846789f8cd-pbccj" Sep 11 20:13:13.210500 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:13.210463 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:13.358988 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:13.358893 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:13.554479 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:13.554439 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:14.354306 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:14.354250 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:15.289594 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:15.289547 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:15.542654 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:15.542558 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:15.698792 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:15.698757 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:15.792460 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:15.792407 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:15.977096 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:15.977048 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:16.424555 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:16.424515 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:17.106376 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:17.106344 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-46da77-8b9b66cc9-zzk5q" Sep 11 20:13:17.120097 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:17.120067 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:17.139298 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:17.139252 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-8e5fddd7-846789f8cd-pbccj" podStartSLOduration=6.139238177 podStartE2EDuration="6.139238177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:13:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:13:13.138110147 +0000 UTC m=+763.186160505" watchObservedRunningTime="2026-09-11 20:13:17.139238177 +0000 UTC m=+767.187288520" Sep 11 20:13:17.219194 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:17.219158 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:17.507562 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:17.507471 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:17.687819 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:17.687786 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:18.074900 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:18.074804 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:18.112561 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:18.112532 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e5fddd7-67d6795d64-d4cwk" Sep 11 20:13:18.139517 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:18.139490 2546 generic.go:356] "Generic (PLEG): container finished" podID="7947d808-ad70-4b13-b396-4936f2588e44" containerID="8ae7fc1b8494c0d7e86d8740688a9e8f3391285332a2888fb1be74972b40716d" exitCode=0 Sep 11 20:13:18.139722 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:18.139562 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs" event={"ID":"7947d808-ad70-4b13-b396-4936f2588e44","Type":"ContainerDied","Data":"8ae7fc1b8494c0d7e86d8740688a9e8f3391285332a2888fb1be74972b40716d"} Sep 11 20:13:18.139999 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:18.139984 2546 scope.go:117] "RemoveContainer" containerID="8ae7fc1b8494c0d7e86d8740688a9e8f3391285332a2888fb1be74972b40716d" Sep 11 20:13:18.141472 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:18.141436 2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 20:13:18.141592 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:18.141523 2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 20:13:18.141664 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:18.141642 2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 20:13:18.171453 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:18.171423 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:18.272689 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:18.272651 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:18.539199 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:18.539166 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:18.640141 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:18.640102 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:19.148204 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:19.148172 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs" event={"ID":"7947d808-ad70-4b13-b396-4936f2588e44","Type":"ContainerStarted","Data":"f99c815f2928ef5c20d2d309eb2d3ea160a8fadac65a37cb05028b5523727a8f"} Sep 11 20:13:19.212423 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:19.212382 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs" Sep 11 20:13:19.212679 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:19.212657 2546 patch_prober.go:28] interesting pod/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Sep 11 20:13:19.212733 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:19.212709 2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs" podUID="7947d808-ad70-4b13-b396-4936f2588e44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Sep 11 20:13:19.468895 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:19.468809 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:20.212539 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:20.212508 2546 patch_prober.go:28] interesting pod/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Sep 11 20:13:20.212916 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:20.212559 2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs" podUID="7947d808-ad70-4b13-b396-4936f2588e44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Sep 11 20:13:20.293127 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:20.293080 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:20.550377 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:20.550293 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:21.212294 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:21.212261 2546 patch_prober.go:28] interesting pod/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Sep 11 20:13:21.212461 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:21.212327 2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs" podUID="7947d808-ad70-4b13-b396-4936f2588e44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Sep 11 20:13:21.465903 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:21.465825 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:21.559041 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:21.559005 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:22.115134 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:22.115105 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:22.205722 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:22.205690 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:22.210375 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:22.210352 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4f9558-6d94fbdbc8-x6znq"] Sep 11 20:13:22.213051 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:22.213020 2546 patch_prober.go:28] interesting pod/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Sep 11 20:13:22.213157 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:22.213080 2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs" podUID="7947d808-ad70-4b13-b396-4936f2588e44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Sep 11 20:13:22.214321 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:22.214303 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4f9558-6d94fbdbc8-x6znq" Sep 11 20:13:22.216263 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:22.216244 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-4f9558\"" Sep 11 20:13:22.247067 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:22.247034 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4f9558-6d94fbdbc8-x6znq"] Sep 11 20:13:22.302379 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:22.302343 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmsgz\" (UniqueName: \"kubernetes.io/projected/19be5c5d-a319-43c1-bc08-976f128fdccc-kube-api-access-dmsgz\") pod \"payload-pre-processing-e2e-inf-b-4f9558-6d94fbdbc8-x6znq\" (UID: \"19be5c5d-a319-43c1-bc08-976f128fdccc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4f9558-6d94fbdbc8-x6znq" Sep 11 20:13:22.302536 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:22.302386 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19be5c5d-a319-43c1-bc08-976f128fdccc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4f9558-6d94fbdbc8-x6znq\" (UID: \"19be5c5d-a319-43c1-bc08-976f128fdccc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4f9558-6d94fbdbc8-x6znq" Sep 11 20:13:22.403119 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:22.403022 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19be5c5d-a319-43c1-bc08-976f128fdccc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4f9558-6d94fbdbc8-x6znq\" (UID: \"19be5c5d-a319-43c1-bc08-976f128fdccc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4f9558-6d94fbdbc8-x6znq" Sep 11 20:13:22.403287 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:22.403224 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmsgz\" (UniqueName: \"kubernetes.io/projected/19be5c5d-a319-43c1-bc08-976f128fdccc-kube-api-access-dmsgz\") pod \"payload-pre-processing-e2e-inf-b-4f9558-6d94fbdbc8-x6znq\" (UID: \"19be5c5d-a319-43c1-bc08-976f128fdccc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4f9558-6d94fbdbc8-x6znq" Sep 11 20:13:22.403927 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:22.403897 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19be5c5d-a319-43c1-bc08-976f128fdccc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4f9558-6d94fbdbc8-x6znq\" (UID: \"19be5c5d-a319-43c1-bc08-976f128fdccc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4f9558-6d94fbdbc8-x6znq" Sep 11 20:13:22.410448 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:22.410424 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmsgz\" (UniqueName: \"kubernetes.io/projected/19be5c5d-a319-43c1-bc08-976f128fdccc-kube-api-access-dmsgz\") pod \"payload-pre-processing-e2e-inf-b-4f9558-6d94fbdbc8-x6znq\" (UID: \"19be5c5d-a319-43c1-bc08-976f128fdccc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4f9558-6d94fbdbc8-x6znq" Sep 11 20:13:22.557801 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:22.557766 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:22.558615 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:22.558589 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4f9558-6d94fbdbc8-x6znq" Sep 11 20:13:22.843600 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:22.839856 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4f9558-6d94fbdbc8-x6znq"] Sep 11 20:13:22.852260 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:22.851906 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:23.174303 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:23.174135 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4f9558-6d94fbdbc8-x6znq" event={"ID":"19be5c5d-a319-43c1-bc08-976f128fdccc","Type":"ContainerStarted","Data":"5125f3b52c0387281d8c8a2669627035e1934872e6f5b53cf4406588906ba21a"} Sep 11 20:13:23.174303 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:23.174202 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4f9558-6d94fbdbc8-x6znq" event={"ID":"19be5c5d-a319-43c1-bc08-976f128fdccc","Type":"ContainerStarted","Data":"ba6beac6285f8cc457cab10bf7e7cbc8befb618ec8983181dfeb475000eab4fc"} Sep 11 20:13:23.174303 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:23.174269 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4f9558-6d94fbdbc8-x6znq" Sep 11 20:13:23.198439 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:23.198388 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4f9558-6d94fbdbc8-x6znq" podStartSLOduration=1.19837183 podStartE2EDuration="1.19837183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:13:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:13:23.195687167 +0000 UTC m=+773.243737511" watchObservedRunningTime="2026-09-11 20:13:23.19837183 +0000 UTC m=+773.246422210" Sep 11 20:13:23.212325 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:23.212292 2546 patch_prober.go:28] interesting pod/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Sep 11 20:13:23.212467 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:23.212338 2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs" podUID="7947d808-ad70-4b13-b396-4936f2588e44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Sep 11 20:13:23.294974 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:23.294929 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:23.390040 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:23.390008 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:23.492792 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:23.492716 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:23.606784 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:23.606753 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:23.829093 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:23.828955 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:23.919774 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:23.919734 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:24.020726 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:24.020689 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:24.128601 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:24.124751 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:24.212516 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:24.212486 2546 patch_prober.go:28] interesting pod/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Sep 11 20:13:24.212713 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:24.212539 2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs" podUID="7947d808-ad70-4b13-b396-4936f2588e44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Sep 11 20:13:24.916904 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:24.916868 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:25.122277 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:25.122246 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8e5fddd7-846789f8cd-pbccj" Sep 11 20:13:25.216284 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:25.216212 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs" Sep 11 20:13:25.216523 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:25.216503 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs" Sep 11 20:13:25.217234 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:25.217217 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs" Sep 11 20:13:25.889813 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:25.889784 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:26.502422 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:26.502383 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:26.724701 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:26.724664 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:27.513081 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:27.513045 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:28.195934 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:28.195901 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:28.932946 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:28.932914 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:29.618066 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:29.618037 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:30.412457 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:30.412422 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:30.573754 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:30.573724 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:30.725286 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:30.725197 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:30.827366 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:30.827330 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:30.929415 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:30.929386 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:31.461599 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:31.461547 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:32.202798 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:32.202760 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:33.024772 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:33.024733 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:33.797751 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:33.797700 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:34.182737 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:34.182707 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4f9558-6d94fbdbc8-x6znq" Sep 11 20:13:37.013996 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:37.013967 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8e5fddd7-67d6795d64-d4cwk"] Sep 11 20:13:37.014443 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:37.014174 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e5fddd7-67d6795d64-d4cwk" podUID="c85c1bd1-4890-4653-ab61-cda78116644d" containerName="maas-api" containerID="cri-o://bb76d27c90d069cdadaf1c03a4db7fbadff5f55aa44fbea005944eab24c7f625" gracePeriod=30 Sep 11 20:13:37.044780 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:37.044745 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:37.080564 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:37.080530 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8e5fddd7-846789f8cd-pbccj"] Sep 11 20:13:37.080813 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:37.080781 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-8e5fddd7-846789f8cd-pbccj" podUID="579d1e65-a172-4b48-b328-1eece36b3a08" containerName="payload-pre-processing" containerID="cri-o://47be490458af7a17a4b5b2a82e1ec9a3cbcf7022d74e7e7bae08dd71ffb36ed6" gracePeriod=30 Sep 11 20:13:37.141995 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:37.141969 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:37.243183 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:37.243094 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:37.266399 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:37.266335 2546 generic.go:356] "Generic (PLEG): container finished" podID="579d1e65-a172-4b48-b328-1eece36b3a08" containerID="47be490458af7a17a4b5b2a82e1ec9a3cbcf7022d74e7e7bae08dd71ffb36ed6" exitCode=0 Sep 11 20:13:37.266532 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:37.266433 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8e5fddd7-846789f8cd-pbccj" event={"ID":"579d1e65-a172-4b48-b328-1eece36b3a08","Type":"ContainerDied","Data":"47be490458af7a17a4b5b2a82e1ec9a3cbcf7022d74e7e7bae08dd71ffb36ed6"} Sep 11 20:13:37.270026 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:37.269879 2546 generic.go:356] "Generic (PLEG): container finished" podID="c85c1bd1-4890-4653-ab61-cda78116644d" containerID="bb76d27c90d069cdadaf1c03a4db7fbadff5f55aa44fbea005944eab24c7f625" exitCode=0 Sep 11 20:13:37.270026 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:37.269962 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e5fddd7-67d6795d64-d4cwk" event={"ID":"c85c1bd1-4890-4653-ab61-cda78116644d","Type":"ContainerDied","Data":"bb76d27c90d069cdadaf1c03a4db7fbadff5f55aa44fbea005944eab24c7f625"} Sep 11 20:13:37.345144 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:37.345098 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:37.352629 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:37.352327 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e5fddd7-67d6795d64-d4cwk" Sep 11 20:13:37.394183 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:37.394159 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8e5fddd7-846789f8cd-pbccj" Sep 11 20:13:37.423492 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:37.423458 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:37.471690 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:37.471657 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c85c1bd1-4890-4653-ab61-cda78116644d-maas-api-tls\") pod \"c85c1bd1-4890-4653-ab61-cda78116644d\" (UID: \"c85c1bd1-4890-4653-ab61-cda78116644d\") " Sep 11 20:13:37.471846 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:37.471710 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c85c1bd1-4890-4653-ab61-cda78116644d-openshift-service-ca\") pod \"c85c1bd1-4890-4653-ab61-cda78116644d\" (UID: \"c85c1bd1-4890-4653-ab61-cda78116644d\") " Sep 11 20:13:37.471846 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:37.471793 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c85c1bd1-4890-4653-ab61-cda78116644d-metrics-certs\") pod \"c85c1bd1-4890-4653-ab61-cda78116644d\" (UID: \"c85c1bd1-4890-4653-ab61-cda78116644d\") " Sep 11 20:13:37.471967 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:37.471864 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l9fnm\" (UniqueName: \"kubernetes.io/projected/c85c1bd1-4890-4653-ab61-cda78116644d-kube-api-access-l9fnm\") pod \"c85c1bd1-4890-4653-ab61-cda78116644d\" (UID: \"c85c1bd1-4890-4653-ab61-cda78116644d\") " Sep 11 20:13:37.472075 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:37.472052 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c85c1bd1-4890-4653-ab61-cda78116644d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c85c1bd1-4890-4653-ab61-cda78116644d" (UID: "c85c1bd1-4890-4653-ab61-cda78116644d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:13:37.472341 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:37.472313 2546 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c85c1bd1-4890-4653-ab61-cda78116644d-openshift-service-ca\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:13:37.473918 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:37.473892 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c85c1bd1-4890-4653-ab61-cda78116644d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c85c1bd1-4890-4653-ab61-cda78116644d" (UID: "c85c1bd1-4890-4653-ab61-cda78116644d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 20:13:37.474069 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:37.474048 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c85c1bd1-4890-4653-ab61-cda78116644d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c85c1bd1-4890-4653-ab61-cda78116644d" (UID: "c85c1bd1-4890-4653-ab61-cda78116644d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 20:13:37.474140 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:37.474060 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c85c1bd1-4890-4653-ab61-cda78116644d-kube-api-access-l9fnm" (OuterVolumeSpecName: "kube-api-access-l9fnm") pod "c85c1bd1-4890-4653-ab61-cda78116644d" (UID: "c85c1bd1-4890-4653-ab61-cda78116644d"). InnerVolumeSpecName "kube-api-access-l9fnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:13:37.573053 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:37.572974 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/579d1e65-a172-4b48-b328-1eece36b3a08-plugins-config-volume\") pod \"579d1e65-a172-4b48-b328-1eece36b3a08\" (UID: \"579d1e65-a172-4b48-b328-1eece36b3a08\") " Sep 11 20:13:37.573227 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:37.573073 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v7xz2\" (UniqueName: \"kubernetes.io/projected/579d1e65-a172-4b48-b328-1eece36b3a08-kube-api-access-v7xz2\") pod \"579d1e65-a172-4b48-b328-1eece36b3a08\" (UID: \"579d1e65-a172-4b48-b328-1eece36b3a08\") " Sep 11 20:13:37.573365 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:37.573341 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/579d1e65-a172-4b48-b328-1eece36b3a08-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "579d1e65-a172-4b48-b328-1eece36b3a08" (UID: "579d1e65-a172-4b48-b328-1eece36b3a08"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:13:37.573445 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:37.573350 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9fnm\" (UniqueName: \"kubernetes.io/projected/c85c1bd1-4890-4653-ab61-cda78116644d-kube-api-access-l9fnm\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:13:37.573445 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:37.573394 2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c85c1bd1-4890-4653-ab61-cda78116644d-maas-api-tls\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:13:37.573445 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:37.573405 2546 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c85c1bd1-4890-4653-ab61-cda78116644d-metrics-certs\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:13:37.575010 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:37.574986 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/579d1e65-a172-4b48-b328-1eece36b3a08-kube-api-access-v7xz2" (OuterVolumeSpecName: "kube-api-access-v7xz2") pod "579d1e65-a172-4b48-b328-1eece36b3a08" (UID: "579d1e65-a172-4b48-b328-1eece36b3a08"). InnerVolumeSpecName "kube-api-access-v7xz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:13:37.674819 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:37.674785 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7xz2\" (UniqueName: \"kubernetes.io/projected/579d1e65-a172-4b48-b328-1eece36b3a08-kube-api-access-v7xz2\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:13:37.674819 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:37.674814 2546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/579d1e65-a172-4b48-b328-1eece36b3a08-plugins-config-volume\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:13:37.679296 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:37.679270 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:37.781742 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:37.781704 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:37.879782 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:37.879741 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:37.982103 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:37.982075 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:38.079744 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:38.079706 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:38.280700 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:38.280623 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8e5fddd7-846789f8cd-pbccj" Sep 11 20:13:38.280700 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:38.280641 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8e5fddd7-846789f8cd-pbccj" event={"ID":"579d1e65-a172-4b48-b328-1eece36b3a08","Type":"ContainerDied","Data":"1edae705445fe46b60f596ec2da34de38f1585faff1a161255f0425ef0735b64"} Sep 11 20:13:38.280929 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:38.280715 2546 scope.go:117] "RemoveContainer" containerID="47be490458af7a17a4b5b2a82e1ec9a3cbcf7022d74e7e7bae08dd71ffb36ed6" Sep 11 20:13:38.282386 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:38.282367 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e5fddd7-67d6795d64-d4cwk" Sep 11 20:13:38.282482 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:38.282390 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e5fddd7-67d6795d64-d4cwk" event={"ID":"c85c1bd1-4890-4653-ab61-cda78116644d","Type":"ContainerDied","Data":"814d25898082ffafd2ffbaa65dcc95cf9a33d93bdb3af9cfa6333f508d41b0ca"} Sep 11 20:13:38.291996 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:38.291978 2546 scope.go:117] "RemoveContainer" containerID="bb76d27c90d069cdadaf1c03a4db7fbadff5f55aa44fbea005944eab24c7f625" Sep 11 20:13:38.312635 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:38.312615 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8e5fddd7-67d6795d64-d4cwk"] Sep 11 20:13:38.319523 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:38.319498 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8e5fddd7-67d6795d64-d4cwk"] Sep 11 20:13:38.332865 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:38.332844 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8e5fddd7-846789f8cd-pbccj"] Sep 11 20:13:38.341785 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:38.341760 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8e5fddd7-846789f8cd-pbccj"] Sep 11 20:13:38.359965 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:38.359942 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt"] Sep 11 20:13:38.360662 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:38.360610 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c85c1bd1-4890-4653-ab61-cda78116644d" containerName="maas-api" Sep 11 20:13:38.360662 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:38.360638 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="c85c1bd1-4890-4653-ab61-cda78116644d" containerName="maas-api" Sep 11 20:13:38.360820 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:38.360683 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="579d1e65-a172-4b48-b328-1eece36b3a08" containerName="payload-pre-processing" Sep 11 20:13:38.360820 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:38.360696 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="579d1e65-a172-4b48-b328-1eece36b3a08" containerName="payload-pre-processing" Sep 11 20:13:38.360820 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:38.360813 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="c85c1bd1-4890-4653-ab61-cda78116644d" containerName="maas-api" Sep 11 20:13:38.360994 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:38.360828 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="579d1e65-a172-4b48-b328-1eece36b3a08" containerName="payload-pre-processing" Sep 11 20:13:38.366150 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:38.366130 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt" Sep 11 20:13:38.369216 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:38.369200 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-c83b7433-gw-openshift-default-dockercfg-kd4jg\"" Sep 11 20:13:38.381651 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:38.381632 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt"] Sep 11 20:13:38.416527 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:38.416496 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:38.454669 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:38.454637 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="579d1e65-a172-4b48-b328-1eece36b3a08" path="/var/lib/kubelet/pods/579d1e65-a172-4b48-b328-1eece36b3a08/volumes" Sep 11 20:13:38.455067 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:38.455053 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c85c1bd1-4890-4653-ab61-cda78116644d" path="/var/lib/kubelet/pods/c85c1bd1-4890-4653-ab61-cda78116644d/volumes" Sep 11 20:13:38.481014 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:38.480969 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7151a8f6-a052-423a-a5be-aec5eec623d0-workload-socket\") pod \"e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt\" (UID: \"7151a8f6-a052-423a-a5be-aec5eec623d0\") " pod="openshift-ingress/e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt" Sep 11 20:13:38.481196 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:38.481081 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7151a8f6-a052-423a-a5be-aec5eec623d0-istio-data\") pod \"e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt\" (UID: \"7151a8f6-a052-423a-a5be-aec5eec623d0\") " pod="openshift-ingress/e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt" Sep 11 20:13:38.481196 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:38.481153 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7151a8f6-a052-423a-a5be-aec5eec623d0-istio-envoy\") pod \"e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt\" (UID: \"7151a8f6-a052-423a-a5be-aec5eec623d0\") " pod="openshift-ingress/e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt" Sep 11 20:13:38.481196 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:38.481193 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7151a8f6-a052-423a-a5be-aec5eec623d0-workload-certs\") pod \"e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt\" (UID: \"7151a8f6-a052-423a-a5be-aec5eec623d0\") " pod="openshift-ingress/e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt" Sep 11 20:13:38.481364 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:38.481213 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7151a8f6-a052-423a-a5be-aec5eec623d0-istiod-ca-cert\") pod \"e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt\" (UID: \"7151a8f6-a052-423a-a5be-aec5eec623d0\") " pod="openshift-ingress/e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt" Sep 11 20:13:38.481364 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:38.481255 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8fv2\" (UniqueName: \"kubernetes.io/projected/7151a8f6-a052-423a-a5be-aec5eec623d0-kube-api-access-v8fv2\") pod \"e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt\" (UID: \"7151a8f6-a052-423a-a5be-aec5eec623d0\") " pod="openshift-ingress/e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt" Sep 11 20:13:38.481364 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:38.481284 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7151a8f6-a052-423a-a5be-aec5eec623d0-istio-podinfo\") pod \"e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt\" (UID: \"7151a8f6-a052-423a-a5be-aec5eec623d0\") " pod="openshift-ingress/e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt" Sep 11 20:13:38.481364 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:38.481313 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7151a8f6-a052-423a-a5be-aec5eec623d0-istio-token\") pod \"e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt\" (UID: \"7151a8f6-a052-423a-a5be-aec5eec623d0\") " pod="openshift-ingress/e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt" Sep 11 20:13:38.481599 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:38.481398 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7151a8f6-a052-423a-a5be-aec5eec623d0-credential-socket\") pod \"e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt\" (UID: \"7151a8f6-a052-423a-a5be-aec5eec623d0\") " pod="openshift-ingress/e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt" Sep 11 20:13:38.582871 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:38.582795 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7151a8f6-a052-423a-a5be-aec5eec623d0-istio-token\") pod \"e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt\" (UID: \"7151a8f6-a052-423a-a5be-aec5eec623d0\") " pod="openshift-ingress/e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt" Sep 11 20:13:38.582871 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:38.582855 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7151a8f6-a052-423a-a5be-aec5eec623d0-credential-socket\") pod \"e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt\" (UID: \"7151a8f6-a052-423a-a5be-aec5eec623d0\") " pod="openshift-ingress/e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt" Sep 11 20:13:38.583087 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:38.582892 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7151a8f6-a052-423a-a5be-aec5eec623d0-workload-socket\") pod \"e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt\" (UID: \"7151a8f6-a052-423a-a5be-aec5eec623d0\") " pod="openshift-ingress/e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt" Sep 11 20:13:38.583087 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:38.582932 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7151a8f6-a052-423a-a5be-aec5eec623d0-istio-data\") pod \"e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt\" (UID: \"7151a8f6-a052-423a-a5be-aec5eec623d0\") " pod="openshift-ingress/e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt" Sep 11 20:13:38.583087 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:38.582982 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7151a8f6-a052-423a-a5be-aec5eec623d0-istio-envoy\") pod \"e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt\" (UID: \"7151a8f6-a052-423a-a5be-aec5eec623d0\") " pod="openshift-ingress/e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt" Sep 11 20:13:38.583087 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:38.583026 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7151a8f6-a052-423a-a5be-aec5eec623d0-workload-certs\") pod \"e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt\" (UID: \"7151a8f6-a052-423a-a5be-aec5eec623d0\") " pod="openshift-ingress/e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt" Sep 11 20:13:38.583087 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:38.583059 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7151a8f6-a052-423a-a5be-aec5eec623d0-istiod-ca-cert\") pod \"e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt\" (UID: \"7151a8f6-a052-423a-a5be-aec5eec623d0\") " pod="openshift-ingress/e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt" Sep 11 20:13:38.583350 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:38.583099 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8fv2\" (UniqueName: \"kubernetes.io/projected/7151a8f6-a052-423a-a5be-aec5eec623d0-kube-api-access-v8fv2\") pod \"e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt\" (UID: \"7151a8f6-a052-423a-a5be-aec5eec623d0\") " pod="openshift-ingress/e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt" Sep 11 20:13:38.583350 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:38.583131 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7151a8f6-a052-423a-a5be-aec5eec623d0-istio-podinfo\") pod \"e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt\" (UID: \"7151a8f6-a052-423a-a5be-aec5eec623d0\") " pod="openshift-ingress/e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt" Sep 11 20:13:38.583350 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:38.583325 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7151a8f6-a052-423a-a5be-aec5eec623d0-credential-socket\") pod \"e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt\" (UID: \"7151a8f6-a052-423a-a5be-aec5eec623d0\") " pod="openshift-ingress/e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt" Sep 11 20:13:38.583464 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:38.583411 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7151a8f6-a052-423a-a5be-aec5eec623d0-workload-socket\") pod \"e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt\" (UID: \"7151a8f6-a052-423a-a5be-aec5eec623d0\") " pod="openshift-ingress/e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt" Sep 11 20:13:38.583607 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:38.583562 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7151a8f6-a052-423a-a5be-aec5eec623d0-workload-certs\") pod \"e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt\" (UID: \"7151a8f6-a052-423a-a5be-aec5eec623d0\") " pod="openshift-ingress/e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt" Sep 11 20:13:38.583697 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:38.583675 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7151a8f6-a052-423a-a5be-aec5eec623d0-istio-data\") pod \"e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt\" (UID: \"7151a8f6-a052-423a-a5be-aec5eec623d0\") " pod="openshift-ingress/e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt" Sep 11 20:13:38.583884 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:38.583867 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7151a8f6-a052-423a-a5be-aec5eec623d0-istiod-ca-cert\") pod \"e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt\" (UID: \"7151a8f6-a052-423a-a5be-aec5eec623d0\") " pod="openshift-ingress/e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt" Sep 11 20:13:38.585373 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:38.585350 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7151a8f6-a052-423a-a5be-aec5eec623d0-istio-envoy\") pod \"e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt\" (UID: \"7151a8f6-a052-423a-a5be-aec5eec623d0\") " pod="openshift-ingress/e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt" Sep 11 20:13:38.585474 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:38.585456 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7151a8f6-a052-423a-a5be-aec5eec623d0-istio-podinfo\") pod \"e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt\" (UID: \"7151a8f6-a052-423a-a5be-aec5eec623d0\") " pod="openshift-ingress/e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt" Sep 11 20:13:38.592863 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:38.592838 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8fv2\" (UniqueName: \"kubernetes.io/projected/7151a8f6-a052-423a-a5be-aec5eec623d0-kube-api-access-v8fv2\") pod \"e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt\" (UID: \"7151a8f6-a052-423a-a5be-aec5eec623d0\") " pod="openshift-ingress/e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt" Sep 11 20:13:38.594158 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:38.594137 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7151a8f6-a052-423a-a5be-aec5eec623d0-istio-token\") pod \"e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt\" (UID: \"7151a8f6-a052-423a-a5be-aec5eec623d0\") " pod="openshift-ingress/e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt" Sep 11 20:13:38.676543 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:38.676520 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt" Sep 11 20:13:38.696020 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:38.695991 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:38.966165 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:38.966142 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt"] Sep 11 20:13:38.967599 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:13:38.967560 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7151a8f6_a052_423a_a5be_aec5eec623d0.slice/crio-64c8c5eeb8938a42f0ca0ffd3392e1253e6d31690cc0c616caa11b1f86a040c8 WatchSource:0}: Error finding container 64c8c5eeb8938a42f0ca0ffd3392e1253e6d31690cc0c616caa11b1f86a040c8: Status 404 returned error can't find the container with id 64c8c5eeb8938a42f0ca0ffd3392e1253e6d31690cc0c616caa11b1f86a040c8 Sep 11 20:13:38.969935 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:38.969901 2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 20:13:38.970024 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:38.969965 2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 20:13:38.970024 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:38.969995 2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 20:13:39.127138 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:39.127106 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:39.290852 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:39.290760 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt" event={"ID":"7151a8f6-a052-423a-a5be-aec5eec623d0","Type":"ContainerStarted","Data":"384c4d729b5836c6f56123b4402a143007ec0e1f29c6873b3663cdaeaeb5f4ec"} Sep 11 20:13:39.290852 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:39.290818 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt" event={"ID":"7151a8f6-a052-423a-a5be-aec5eec623d0","Type":"ContainerStarted","Data":"64c8c5eeb8938a42f0ca0ffd3392e1253e6d31690cc0c616caa11b1f86a040c8"} Sep 11 20:13:39.313565 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:39.313518 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt" podStartSLOduration=1.313500482 podStartE2EDuration="1.313500482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:13:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:13:39.311745144 +0000 UTC m=+789.359795487" watchObservedRunningTime="2026-09-11 20:13:39.313500482 +0000 UTC m=+789.361550828" Sep 11 20:13:39.619079 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:39.619045 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:39.677119 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:39.677089 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt" Sep 11 20:13:40.682310 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:40.682282 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt" Sep 11 20:13:40.792888 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:40.792858 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:40.915602 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:40.915558 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:41.014388 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:41.014313 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:41.303434 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:41.303349 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt" Sep 11 20:13:41.304415 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:41.304395 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt" Sep 11 20:13:41.658717 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:41.658689 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:41.845892 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:41.845861 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:42.474631 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:42.474596 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:43.243841 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:43.243803 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:43.344432 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:43.344397 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:43.440251 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:43.440216 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:43.706856 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:43.706829 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:43.804210 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:43.804174 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:43.852014 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:43.851982 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c83b7433-7456fd9966-z6lwc"] Sep 11 20:13:43.856777 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:43.856754 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c83b7433-7456fd9966-z6lwc" Sep 11 20:13:43.858982 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:43.858961 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-c83b7433-dockercfg-25mmr\"" Sep 11 20:13:43.859089 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:43.858972 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-c83b7433\"" Sep 11 20:13:43.863767 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:43.863742 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c83b7433-7456fd9966-z6lwc"] Sep 11 20:13:44.036523 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:44.036429 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f724d7a3-7e4c-46cb-829d-c4ec11632cac-plugins-config-volume\") pod \"payload-processing-e2e-migrate-c83b7433-7456fd9966-z6lwc\" (UID: \"f724d7a3-7e4c-46cb-829d-c4ec11632cac\") " pod="openshift-ingress/payload-processing-e2e-migrate-c83b7433-7456fd9966-z6lwc" Sep 11 20:13:44.036523 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:44.036477 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfj4m\" (UniqueName: \"kubernetes.io/projected/f724d7a3-7e4c-46cb-829d-c4ec11632cac-kube-api-access-jfj4m\") pod \"payload-processing-e2e-migrate-c83b7433-7456fd9966-z6lwc\" (UID: \"f724d7a3-7e4c-46cb-829d-c4ec11632cac\") " pod="openshift-ingress/payload-processing-e2e-migrate-c83b7433-7456fd9966-z6lwc" Sep 11 20:13:44.110567 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:44.110530 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:44.137738 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:44.137706 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f724d7a3-7e4c-46cb-829d-c4ec11632cac-plugins-config-volume\") pod \"payload-processing-e2e-migrate-c83b7433-7456fd9966-z6lwc\" (UID: \"f724d7a3-7e4c-46cb-829d-c4ec11632cac\") " pod="openshift-ingress/payload-processing-e2e-migrate-c83b7433-7456fd9966-z6lwc" Sep 11 20:13:44.137892 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:44.137748 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfj4m\" (UniqueName: \"kubernetes.io/projected/f724d7a3-7e4c-46cb-829d-c4ec11632cac-kube-api-access-jfj4m\") pod \"payload-processing-e2e-migrate-c83b7433-7456fd9966-z6lwc\" (UID: \"f724d7a3-7e4c-46cb-829d-c4ec11632cac\") " pod="openshift-ingress/payload-processing-e2e-migrate-c83b7433-7456fd9966-z6lwc" Sep 11 20:13:44.138368 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:44.138347 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f724d7a3-7e4c-46cb-829d-c4ec11632cac-plugins-config-volume\") pod \"payload-processing-e2e-migrate-c83b7433-7456fd9966-z6lwc\" (UID: \"f724d7a3-7e4c-46cb-829d-c4ec11632cac\") " pod="openshift-ingress/payload-processing-e2e-migrate-c83b7433-7456fd9966-z6lwc" Sep 11 20:13:44.145756 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:44.145736 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfj4m\" (UniqueName: \"kubernetes.io/projected/f724d7a3-7e4c-46cb-829d-c4ec11632cac-kube-api-access-jfj4m\") pod \"payload-processing-e2e-migrate-c83b7433-7456fd9966-z6lwc\" (UID: \"f724d7a3-7e4c-46cb-829d-c4ec11632cac\") " pod="openshift-ingress/payload-processing-e2e-migrate-c83b7433-7456fd9966-z6lwc" Sep 11 20:13:44.169518 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:44.169496 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c83b7433-7456fd9966-z6lwc" Sep 11 20:13:44.452221 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:13:44.452193 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf724d7a3_7e4c_46cb_829d_c4ec11632cac.slice/crio-656bffe6d5431af77186791ceb942d2eb9172c59185aff069310b47ecaf9a7ff WatchSource:0}: Error finding container 656bffe6d5431af77186791ceb942d2eb9172c59185aff069310b47ecaf9a7ff: Status 404 returned error can't find the container with id 656bffe6d5431af77186791ceb942d2eb9172c59185aff069310b47ecaf9a7ff Sep 11 20:13:44.452221 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:44.452217 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c83b7433-7456fd9966-z6lwc"] Sep 11 20:13:45.017688 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:45.017655 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:45.329938 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:45.329857 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c83b7433-7456fd9966-z6lwc" event={"ID":"f724d7a3-7e4c-46cb-829d-c4ec11632cac","Type":"ContainerStarted","Data":"f047814a5bbfac880e04d7aa50c516f5386376d350ba292f586da179b1a71a1a"} Sep 11 20:13:45.329938 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:45.329906 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c83b7433-7456fd9966-z6lwc" event={"ID":"f724d7a3-7e4c-46cb-829d-c4ec11632cac","Type":"ContainerStarted","Data":"656bffe6d5431af77186791ceb942d2eb9172c59185aff069310b47ecaf9a7ff"} Sep 11 20:13:45.330173 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:45.330001 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-c83b7433-7456fd9966-z6lwc" Sep 11 20:13:45.346959 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:45.346901 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-c83b7433-7456fd9966-z6lwc" podStartSLOduration=2.34688422 podStartE2EDuration="2.34688422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:13:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:13:45.34620917 +0000 UTC m=+795.394259514" watchObservedRunningTime="2026-09-11 20:13:45.34688422 +0000 UTC m=+795.394934565" Sep 11 20:13:45.868750 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:45.868721 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:46.603701 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:46.603671 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:47.294870 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:47.294836 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:56.338068 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:56.338039 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-c83b7433-7456fd9966-z6lwc" Sep 11 20:13:57.345804 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:57.345765 2546 patch_prober.go:28] interesting pod/e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 11 20:13:57.346299 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:57.345842 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt" podUID="7151a8f6-a052-423a-a5be-aec5eec623d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 11 20:13:59.414539 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:59.414499 2546 generic.go:356] "Generic (PLEG): container finished" podID="7151a8f6-a052-423a-a5be-aec5eec623d0" containerID="384c4d729b5836c6f56123b4402a143007ec0e1f29c6873b3663cdaeaeb5f4ec" exitCode=0 Sep 11 20:13:59.414975 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:59.414571 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt" event={"ID":"7151a8f6-a052-423a-a5be-aec5eec623d0","Type":"ContainerDied","Data":"384c4d729b5836c6f56123b4402a143007ec0e1f29c6873b3663cdaeaeb5f4ec"} Sep 11 20:13:59.415060 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:59.415045 2546 scope.go:117] "RemoveContainer" containerID="384c4d729b5836c6f56123b4402a143007ec0e1f29c6873b3663cdaeaeb5f4ec" Sep 11 20:13:59.416130 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:59.416099 2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 20:13:59.416230 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:59.416190 2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 20:13:59.416294 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:13:59.416238 2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 20:14:00.421915 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:00.421878 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt" event={"ID":"7151a8f6-a052-423a-a5be-aec5eec623d0","Type":"ContainerStarted","Data":"0d7f0c1ff001a6b3ff5f5e55a8326c2a47a5d3d91b7fd8f1f48c0fddd765e711"} Sep 11 20:14:00.677190 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:00.677102 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt" Sep 11 20:14:00.677379 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:00.677357 2546 patch_prober.go:28] interesting pod/e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Sep 11 20:14:00.677429 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:00.677415 2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt" podUID="7151a8f6-a052-423a-a5be-aec5eec623d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Sep 11 20:14:01.677732 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:01.677697 2546 patch_prober.go:28] interesting pod/e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Sep 11 20:14:01.678116 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:01.677752 2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt" podUID="7151a8f6-a052-423a-a5be-aec5eec623d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Sep 11 20:14:02.677940 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:02.677898 2546 patch_prober.go:28] interesting pod/e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Sep 11 20:14:02.678303 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:02.677966 2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt" podUID="7151a8f6-a052-423a-a5be-aec5eec623d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Sep 11 20:14:03.677294 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:03.677255 2546 patch_prober.go:28] interesting pod/e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Sep 11 20:14:03.677454 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:03.677316 2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt" podUID="7151a8f6-a052-423a-a5be-aec5eec623d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Sep 11 20:14:04.677518 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:04.677476 2546 patch_prober.go:28] interesting pod/e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Sep 11 20:14:04.677919 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:04.677533 2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt" podUID="7151a8f6-a052-423a-a5be-aec5eec623d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Sep 11 20:14:05.517648 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:05.517611 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:05.681017 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:05.680984 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt" Sep 11 20:14:05.681401 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:05.681253 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt" Sep 11 20:14:05.681885 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:05.681865 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt" Sep 11 20:14:06.469183 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:06.469152 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:06.684904 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:06.684872 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:07.336109 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:07.336061 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:08.223519 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:08.223487 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:09.021098 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:09.021047 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:09.922791 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:09.922760 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:10.607616 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:10.607564 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:11.429139 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:11.429104 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:12.205395 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:12.205362 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:12.236717 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:12.236683 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c83b7433-7456fd9966-z6lwc"] Sep 11 20:14:12.237104 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:12.237036 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-c83b7433-7456fd9966-z6lwc" podUID="f724d7a3-7e4c-46cb-829d-c4ec11632cac" containerName="payload-processing" containerID="cri-o://f047814a5bbfac880e04d7aa50c516f5386376d350ba292f586da179b1a71a1a" gracePeriod=30 Sep 11 20:14:12.307550 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:12.307515 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:12.407646 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:12.407131 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:12.482994 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:12.482949 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:12.489854 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:12.489831 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c83b7433-7456fd9966-z6lwc" Sep 11 20:14:12.506448 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:12.506415 2546 generic.go:356] "Generic (PLEG): container finished" podID="f724d7a3-7e4c-46cb-829d-c4ec11632cac" containerID="f047814a5bbfac880e04d7aa50c516f5386376d350ba292f586da179b1a71a1a" exitCode=0 Sep 11 20:14:12.506590 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:12.506452 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c83b7433-7456fd9966-z6lwc" Sep 11 20:14:12.506590 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:12.506447 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c83b7433-7456fd9966-z6lwc" event={"ID":"f724d7a3-7e4c-46cb-829d-c4ec11632cac","Type":"ContainerDied","Data":"f047814a5bbfac880e04d7aa50c516f5386376d350ba292f586da179b1a71a1a"} Sep 11 20:14:12.506590 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:12.506535 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c83b7433-7456fd9966-z6lwc" event={"ID":"f724d7a3-7e4c-46cb-829d-c4ec11632cac","Type":"ContainerDied","Data":"656bffe6d5431af77186791ceb942d2eb9172c59185aff069310b47ecaf9a7ff"} Sep 11 20:14:12.506590 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:12.506561 2546 scope.go:117] "RemoveContainer" containerID="f047814a5bbfac880e04d7aa50c516f5386376d350ba292f586da179b1a71a1a" Sep 11 20:14:12.519239 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:12.519212 2546 scope.go:117] "RemoveContainer" containerID="f047814a5bbfac880e04d7aa50c516f5386376d350ba292f586da179b1a71a1a" Sep 11 20:14:12.519625 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:14:12.519570 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f047814a5bbfac880e04d7aa50c516f5386376d350ba292f586da179b1a71a1a\": container with ID starting with f047814a5bbfac880e04d7aa50c516f5386376d350ba292f586da179b1a71a1a not found: ID does not exist" containerID="f047814a5bbfac880e04d7aa50c516f5386376d350ba292f586da179b1a71a1a" Sep 11 20:14:12.519704 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:12.519633 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f047814a5bbfac880e04d7aa50c516f5386376d350ba292f586da179b1a71a1a"} err="failed to get container status \"f047814a5bbfac880e04d7aa50c516f5386376d350ba292f586da179b1a71a1a\": rpc error: code = NotFound desc = could not find container \"f047814a5bbfac880e04d7aa50c516f5386376d350ba292f586da179b1a71a1a\": container with ID starting with f047814a5bbfac880e04d7aa50c516f5386376d350ba292f586da179b1a71a1a not found: ID does not exist" Sep 11 20:14:12.544982 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:12.544947 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f724d7a3-7e4c-46cb-829d-c4ec11632cac-plugins-config-volume\") pod \"f724d7a3-7e4c-46cb-829d-c4ec11632cac\" (UID: \"f724d7a3-7e4c-46cb-829d-c4ec11632cac\") " Sep 11 20:14:12.545387 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:12.545364 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jfj4m\" (UniqueName: \"kubernetes.io/projected/f724d7a3-7e4c-46cb-829d-c4ec11632cac-kube-api-access-jfj4m\") pod \"f724d7a3-7e4c-46cb-829d-c4ec11632cac\" (UID: \"f724d7a3-7e4c-46cb-829d-c4ec11632cac\") " Sep 11 20:14:12.545628 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:12.545370 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f724d7a3-7e4c-46cb-829d-c4ec11632cac-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f724d7a3-7e4c-46cb-829d-c4ec11632cac" (UID: "f724d7a3-7e4c-46cb-829d-c4ec11632cac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:14:12.545917 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:12.545896 2546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f724d7a3-7e4c-46cb-829d-c4ec11632cac-plugins-config-volume\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:14:12.552776 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:12.552752 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt"] Sep 11 20:14:12.553428 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:12.553390 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt" podUID="7151a8f6-a052-423a-a5be-aec5eec623d0" containerName="istio-proxy" containerID="cri-o://0d7f0c1ff001a6b3ff5f5e55a8326c2a47a5d3d91b7fd8f1f48c0fddd765e711" gracePeriod=30 Sep 11 20:14:12.564933 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:12.564900 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f724d7a3-7e4c-46cb-829d-c4ec11632cac-kube-api-access-jfj4m" (OuterVolumeSpecName: "kube-api-access-jfj4m") pod "f724d7a3-7e4c-46cb-829d-c4ec11632cac" (UID: "f724d7a3-7e4c-46cb-829d-c4ec11632cac"). InnerVolumeSpecName "kube-api-access-jfj4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:14:12.570966 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:12.570916 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:12.655602 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:12.652377 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfj4m\" (UniqueName: \"kubernetes.io/projected/f724d7a3-7e4c-46cb-829d-c4ec11632cac-kube-api-access-jfj4m\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:14:12.830816 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:12.830753 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c83b7433-7456fd9966-z6lwc"] Sep 11 20:14:12.833928 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:12.833902 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c83b7433-7456fd9966-z6lwc"] Sep 11 20:14:12.995117 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:12.995072 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:13.096729 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:13.096700 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:13.299711 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:13.299684 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:13.573002 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:13.572910 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:13.933393 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:13.933354 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:14.451493 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:14.451458 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f724d7a3-7e4c-46cb-829d-c4ec11632cac" path="/var/lib/kubelet/pods/f724d7a3-7e4c-46cb-829d-c4ec11632cac/volumes" Sep 11 20:14:14.893127 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:14.893086 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:15.651541 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:15.651503 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:15.745490 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:15.745451 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:16.634193 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:16.634153 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:17.599609 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:17.599565 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:17.803897 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:17.803872 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt" Sep 11 20:14:17.900680 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:17.900658 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7151a8f6-a052-423a-a5be-aec5eec623d0-istio-podinfo\") pod \"7151a8f6-a052-423a-a5be-aec5eec623d0\" (UID: \"7151a8f6-a052-423a-a5be-aec5eec623d0\") " Sep 11 20:14:17.900871 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:17.900687 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7151a8f6-a052-423a-a5be-aec5eec623d0-workload-certs\") pod \"7151a8f6-a052-423a-a5be-aec5eec623d0\" (UID: \"7151a8f6-a052-423a-a5be-aec5eec623d0\") " Sep 11 20:14:17.900871 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:17.900730 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7151a8f6-a052-423a-a5be-aec5eec623d0-istio-data\") pod \"7151a8f6-a052-423a-a5be-aec5eec623d0\" (UID: \"7151a8f6-a052-423a-a5be-aec5eec623d0\") " Sep 11 20:14:17.900871 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:17.900762 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7151a8f6-a052-423a-a5be-aec5eec623d0-workload-socket\") pod \"7151a8f6-a052-423a-a5be-aec5eec623d0\" (UID: \"7151a8f6-a052-423a-a5be-aec5eec623d0\") " Sep 11 20:14:17.900871 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:17.900826 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7151a8f6-a052-423a-a5be-aec5eec623d0-istiod-ca-cert\") pod \"7151a8f6-a052-423a-a5be-aec5eec623d0\" (UID: \"7151a8f6-a052-423a-a5be-aec5eec623d0\") " Sep 11 20:14:17.900871 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:17.900845 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7151a8f6-a052-423a-a5be-aec5eec623d0-istio-token\") pod \"7151a8f6-a052-423a-a5be-aec5eec623d0\" (UID: \"7151a8f6-a052-423a-a5be-aec5eec623d0\") " Sep 11 20:14:17.900871 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:17.900867 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7151a8f6-a052-423a-a5be-aec5eec623d0-credential-socket\") pod \"7151a8f6-a052-423a-a5be-aec5eec623d0\" (UID: \"7151a8f6-a052-423a-a5be-aec5eec623d0\") " Sep 11 20:14:17.901195 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:17.900899 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7151a8f6-a052-423a-a5be-aec5eec623d0-istio-envoy\") pod \"7151a8f6-a052-423a-a5be-aec5eec623d0\" (UID: \"7151a8f6-a052-423a-a5be-aec5eec623d0\") " Sep 11 20:14:17.901195 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:17.900925 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8fv2\" (UniqueName: \"kubernetes.io/projected/7151a8f6-a052-423a-a5be-aec5eec623d0-kube-api-access-v8fv2\") pod \"7151a8f6-a052-423a-a5be-aec5eec623d0\" (UID: \"7151a8f6-a052-423a-a5be-aec5eec623d0\") " Sep 11 20:14:17.901195 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:17.900966 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7151a8f6-a052-423a-a5be-aec5eec623d0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7151a8f6-a052-423a-a5be-aec5eec623d0" (UID: "7151a8f6-a052-423a-a5be-aec5eec623d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:14:17.901195 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:17.901114 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7151a8f6-a052-423a-a5be-aec5eec623d0-istio-data" (OuterVolumeSpecName: "istio-data") pod "7151a8f6-a052-423a-a5be-aec5eec623d0" (UID: "7151a8f6-a052-423a-a5be-aec5eec623d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:14:17.901195 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:17.901156 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7151a8f6-a052-423a-a5be-aec5eec623d0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7151a8f6-a052-423a-a5be-aec5eec623d0" (UID: "7151a8f6-a052-423a-a5be-aec5eec623d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:14:17.901467 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:17.901281 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7151a8f6-a052-423a-a5be-aec5eec623d0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7151a8f6-a052-423a-a5be-aec5eec623d0" (UID: "7151a8f6-a052-423a-a5be-aec5eec623d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:14:17.901467 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:17.901208 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7151a8f6-a052-423a-a5be-aec5eec623d0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7151a8f6-a052-423a-a5be-aec5eec623d0" (UID: "7151a8f6-a052-423a-a5be-aec5eec623d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:14:17.901467 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:17.901367 2546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7151a8f6-a052-423a-a5be-aec5eec623d0-istio-data\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:14:17.901467 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:17.901386 2546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7151a8f6-a052-423a-a5be-aec5eec623d0-workload-socket\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:14:17.901467 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:17.901402 2546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7151a8f6-a052-423a-a5be-aec5eec623d0-istiod-ca-cert\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:14:17.901467 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:17.901414 2546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7151a8f6-a052-423a-a5be-aec5eec623d0-credential-socket\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:14:17.901467 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:17.901423 2546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7151a8f6-a052-423a-a5be-aec5eec623d0-workload-certs\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:14:17.902973 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:17.902946 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7151a8f6-a052-423a-a5be-aec5eec623d0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7151a8f6-a052-423a-a5be-aec5eec623d0" (UID: "7151a8f6-a052-423a-a5be-aec5eec623d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:14:17.903216 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:17.903185 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7151a8f6-a052-423a-a5be-aec5eec623d0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7151a8f6-a052-423a-a5be-aec5eec623d0" (UID: "7151a8f6-a052-423a-a5be-aec5eec623d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 20:14:17.903444 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:17.903418 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7151a8f6-a052-423a-a5be-aec5eec623d0-istio-token" (OuterVolumeSpecName: "istio-token") pod "7151a8f6-a052-423a-a5be-aec5eec623d0" (UID: "7151a8f6-a052-423a-a5be-aec5eec623d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:14:17.903512 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:17.903447 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7151a8f6-a052-423a-a5be-aec5eec623d0-kube-api-access-v8fv2" (OuterVolumeSpecName: "kube-api-access-v8fv2") pod "7151a8f6-a052-423a-a5be-aec5eec623d0" (UID: "7151a8f6-a052-423a-a5be-aec5eec623d0"). InnerVolumeSpecName "kube-api-access-v8fv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:14:18.002653 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:18.002623 2546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7151a8f6-a052-423a-a5be-aec5eec623d0-istio-podinfo\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:14:18.002653 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:18.002649 2546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7151a8f6-a052-423a-a5be-aec5eec623d0-istio-token\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:14:18.002812 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:18.002661 2546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7151a8f6-a052-423a-a5be-aec5eec623d0-istio-envoy\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:14:18.002812 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:18.002676 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8fv2\" (UniqueName: \"kubernetes.io/projected/7151a8f6-a052-423a-a5be-aec5eec623d0-kube-api-access-v8fv2\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:14:18.451606 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:18.451558 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:18.549985 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:18.549955 2546 generic.go:356] "Generic (PLEG): container finished" podID="7151a8f6-a052-423a-a5be-aec5eec623d0" containerID="0d7f0c1ff001a6b3ff5f5e55a8326c2a47a5d3d91b7fd8f1f48c0fddd765e711" exitCode=0 Sep 11 20:14:18.550139 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:18.550004 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt" Sep 11 20:14:18.550139 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:18.550006 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt" event={"ID":"7151a8f6-a052-423a-a5be-aec5eec623d0","Type":"ContainerDied","Data":"0d7f0c1ff001a6b3ff5f5e55a8326c2a47a5d3d91b7fd8f1f48c0fddd765e711"} Sep 11 20:14:18.550139 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:18.550130 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt" event={"ID":"7151a8f6-a052-423a-a5be-aec5eec623d0","Type":"ContainerDied","Data":"64c8c5eeb8938a42f0ca0ffd3392e1253e6d31690cc0c616caa11b1f86a040c8"} Sep 11 20:14:18.550246 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:18.550159 2546 scope.go:117] "RemoveContainer" containerID="0d7f0c1ff001a6b3ff5f5e55a8326c2a47a5d3d91b7fd8f1f48c0fddd765e711" Sep 11 20:14:18.553195 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:18.553175 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:18.559548 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:18.559528 2546 scope.go:117] "RemoveContainer" containerID="384c4d729b5836c6f56123b4402a143007ec0e1f29c6873b3663cdaeaeb5f4ec" Sep 11 20:14:18.569176 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:18.569153 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt"] Sep 11 20:14:18.569802 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:18.569784 2546 scope.go:117] "RemoveContainer" containerID="0d7f0c1ff001a6b3ff5f5e55a8326c2a47a5d3d91b7fd8f1f48c0fddd765e711" Sep 11 20:14:18.570076 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:14:18.570058 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d7f0c1ff001a6b3ff5f5e55a8326c2a47a5d3d91b7fd8f1f48c0fddd765e711\": container with ID starting with 0d7f0c1ff001a6b3ff5f5e55a8326c2a47a5d3d91b7fd8f1f48c0fddd765e711 not found: ID does not exist" containerID="0d7f0c1ff001a6b3ff5f5e55a8326c2a47a5d3d91b7fd8f1f48c0fddd765e711" Sep 11 20:14:18.570134 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:18.570102 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d7f0c1ff001a6b3ff5f5e55a8326c2a47a5d3d91b7fd8f1f48c0fddd765e711"} err="failed to get container status \"0d7f0c1ff001a6b3ff5f5e55a8326c2a47a5d3d91b7fd8f1f48c0fddd765e711\": rpc error: code = NotFound desc = could not find container \"0d7f0c1ff001a6b3ff5f5e55a8326c2a47a5d3d91b7fd8f1f48c0fddd765e711\": container with ID starting with 0d7f0c1ff001a6b3ff5f5e55a8326c2a47a5d3d91b7fd8f1f48c0fddd765e711 not found: ID does not exist" Sep 11 20:14:18.570134 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:18.570120 2546 scope.go:117] "RemoveContainer" containerID="384c4d729b5836c6f56123b4402a143007ec0e1f29c6873b3663cdaeaeb5f4ec" Sep 11 20:14:18.570381 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:14:18.570368 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"384c4d729b5836c6f56123b4402a143007ec0e1f29c6873b3663cdaeaeb5f4ec\": container with ID starting with 384c4d729b5836c6f56123b4402a143007ec0e1f29c6873b3663cdaeaeb5f4ec not found: ID does not exist" containerID="384c4d729b5836c6f56123b4402a143007ec0e1f29c6873b3663cdaeaeb5f4ec" Sep 11 20:14:18.570424 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:18.570383 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"384c4d729b5836c6f56123b4402a143007ec0e1f29c6873b3663cdaeaeb5f4ec"} err="failed to get container status \"384c4d729b5836c6f56123b4402a143007ec0e1f29c6873b3663cdaeaeb5f4ec\": rpc error: code = NotFound desc = could not find container \"384c4d729b5836c6f56123b4402a143007ec0e1f29c6873b3663cdaeaeb5f4ec\": container with ID starting with 384c4d729b5836c6f56123b4402a143007ec0e1f29c6873b3663cdaeaeb5f4ec not found: ID does not exist" Sep 11 20:14:18.571616 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:18.571598 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-c83b7433-gw-openshift-default-587d5548-qpvdt"] Sep 11 20:14:18.727440 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:18.727380 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:18.949715 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:18.949684 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:19.047324 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:19.047250 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:19.144912 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:19.144881 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-16cd58ce-7bc78c664b-zkwb2"] Sep 11 20:14:19.145438 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:19.145418 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7151a8f6-a052-423a-a5be-aec5eec623d0" containerName="istio-proxy" Sep 11 20:14:19.145533 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:19.145441 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="7151a8f6-a052-423a-a5be-aec5eec623d0" containerName="istio-proxy" Sep 11 20:14:19.145533 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:19.145476 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f724d7a3-7e4c-46cb-829d-c4ec11632cac" containerName="payload-processing" Sep 11 20:14:19.145533 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:19.145485 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="f724d7a3-7e4c-46cb-829d-c4ec11632cac" containerName="payload-processing" Sep 11 20:14:19.145711 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:19.145639 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="7151a8f6-a052-423a-a5be-aec5eec623d0" containerName="istio-proxy" Sep 11 20:14:19.145711 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:19.145662 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="f724d7a3-7e4c-46cb-829d-c4ec11632cac" containerName="payload-processing" Sep 11 20:14:19.148757 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:19.148737 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-16cd58ce-7bc78c664b-zkwb2" Sep 11 20:14:19.150913 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:19.150893 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-16cd58ce-dockercfg-kbmh6\"" Sep 11 20:14:19.151003 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:19.150923 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-16cd58ce\"" Sep 11 20:14:19.157142 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:19.157123 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-16cd58ce-7bc78c664b-zkwb2"] Sep 11 20:14:19.214324 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:19.214269 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e382f502-335b-4dbe-9139-152a31cb28fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-16cd58ce-7bc78c664b-zkwb2\" (UID: \"e382f502-335b-4dbe-9139-152a31cb28fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-16cd58ce-7bc78c664b-zkwb2" Sep 11 20:14:19.214530 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:19.214344 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78mh8\" (UniqueName: \"kubernetes.io/projected/e382f502-335b-4dbe-9139-152a31cb28fc-kube-api-access-78mh8\") pod \"payload-pre-processing-e2e-ait-16cd58ce-7bc78c664b-zkwb2\" (UID: \"e382f502-335b-4dbe-9139-152a31cb28fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-16cd58ce-7bc78c664b-zkwb2" Sep 11 20:14:19.232030 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:19.231996 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-16cd58ce-6b68959c7c-2rl4l"] Sep 11 20:14:19.232722 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:19.232704 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7151a8f6-a052-423a-a5be-aec5eec623d0" containerName="istio-proxy" Sep 11 20:14:19.232786 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:19.232725 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="7151a8f6-a052-423a-a5be-aec5eec623d0" containerName="istio-proxy" Sep 11 20:14:19.232823 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:19.232787 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="7151a8f6-a052-423a-a5be-aec5eec623d0" containerName="istio-proxy" Sep 11 20:14:19.235939 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:19.235923 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-16cd58ce-6b68959c7c-2rl4l" Sep 11 20:14:19.246622 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:19.246594 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-16cd58ce-6b68959c7c-2rl4l"] Sep 11 20:14:19.315878 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:19.315802 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxjdc\" (UniqueName: \"kubernetes.io/projected/145bf30f-2722-4c68-ab55-a9d05d03f80b-kube-api-access-sxjdc\") pod \"payload-processing-e2e-ait-16cd58ce-6b68959c7c-2rl4l\" (UID: \"145bf30f-2722-4c68-ab55-a9d05d03f80b\") " pod="openshift-ingress/payload-processing-e2e-ait-16cd58ce-6b68959c7c-2rl4l" Sep 11 20:14:19.315878 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:19.315866 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e382f502-335b-4dbe-9139-152a31cb28fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-16cd58ce-7bc78c664b-zkwb2\" (UID: \"e382f502-335b-4dbe-9139-152a31cb28fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-16cd58ce-7bc78c664b-zkwb2" Sep 11 20:14:19.316104 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:19.315887 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78mh8\" (UniqueName: \"kubernetes.io/projected/e382f502-335b-4dbe-9139-152a31cb28fc-kube-api-access-78mh8\") pod \"payload-pre-processing-e2e-ait-16cd58ce-7bc78c664b-zkwb2\" (UID: \"e382f502-335b-4dbe-9139-152a31cb28fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-16cd58ce-7bc78c664b-zkwb2" Sep 11 20:14:19.316104 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:19.315927 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/145bf30f-2722-4c68-ab55-a9d05d03f80b-plugins-config-volume\") pod \"payload-processing-e2e-ait-16cd58ce-6b68959c7c-2rl4l\" (UID: \"145bf30f-2722-4c68-ab55-a9d05d03f80b\") " pod="openshift-ingress/payload-processing-e2e-ait-16cd58ce-6b68959c7c-2rl4l" Sep 11 20:14:19.316540 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:19.316516 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e382f502-335b-4dbe-9139-152a31cb28fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-16cd58ce-7bc78c664b-zkwb2\" (UID: \"e382f502-335b-4dbe-9139-152a31cb28fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-16cd58ce-7bc78c664b-zkwb2" Sep 11 20:14:19.323507 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:19.323487 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78mh8\" (UniqueName: \"kubernetes.io/projected/e382f502-335b-4dbe-9139-152a31cb28fc-kube-api-access-78mh8\") pod \"payload-pre-processing-e2e-ait-16cd58ce-7bc78c664b-zkwb2\" (UID: \"e382f502-335b-4dbe-9139-152a31cb28fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-16cd58ce-7bc78c664b-zkwb2" Sep 11 20:14:19.418592 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:19.413134 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:19.420091 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:19.420061 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/145bf30f-2722-4c68-ab55-a9d05d03f80b-plugins-config-volume\") pod \"payload-processing-e2e-ait-16cd58ce-6b68959c7c-2rl4l\" (UID: \"145bf30f-2722-4c68-ab55-a9d05d03f80b\") " pod="openshift-ingress/payload-processing-e2e-ait-16cd58ce-6b68959c7c-2rl4l" Sep 11 20:14:19.420214 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:19.420203 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxjdc\" (UniqueName: \"kubernetes.io/projected/145bf30f-2722-4c68-ab55-a9d05d03f80b-kube-api-access-sxjdc\") pod \"payload-processing-e2e-ait-16cd58ce-6b68959c7c-2rl4l\" (UID: \"145bf30f-2722-4c68-ab55-a9d05d03f80b\") " pod="openshift-ingress/payload-processing-e2e-ait-16cd58ce-6b68959c7c-2rl4l" Sep 11 20:14:19.420743 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:19.420723 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/145bf30f-2722-4c68-ab55-a9d05d03f80b-plugins-config-volume\") pod \"payload-processing-e2e-ait-16cd58ce-6b68959c7c-2rl4l\" (UID: \"145bf30f-2722-4c68-ab55-a9d05d03f80b\") " pod="openshift-ingress/payload-processing-e2e-ait-16cd58ce-6b68959c7c-2rl4l" Sep 11 20:14:19.428205 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:19.428186 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxjdc\" (UniqueName: \"kubernetes.io/projected/145bf30f-2722-4c68-ab55-a9d05d03f80b-kube-api-access-sxjdc\") pod \"payload-processing-e2e-ait-16cd58ce-6b68959c7c-2rl4l\" (UID: \"145bf30f-2722-4c68-ab55-a9d05d03f80b\") " pod="openshift-ingress/payload-processing-e2e-ait-16cd58ce-6b68959c7c-2rl4l" Sep 11 20:14:19.460527 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:19.460502 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-16cd58ce-7bc78c664b-zkwb2" Sep 11 20:14:19.548849 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:19.548823 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-16cd58ce-6b68959c7c-2rl4l" Sep 11 20:14:19.740552 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:19.740526 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-16cd58ce-7bc78c664b-zkwb2"] Sep 11 20:14:19.742478 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:14:19.742452 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode382f502_335b_4dbe_9139_152a31cb28fc.slice/crio-c3aa5a60c732b03e6723c83742581a554bf4e9d9fda22a4e91a3547962d33261 WatchSource:0}: Error finding container c3aa5a60c732b03e6723c83742581a554bf4e9d9fda22a4e91a3547962d33261: Status 404 returned error can't find the container with id c3aa5a60c732b03e6723c83742581a554bf4e9d9fda22a4e91a3547962d33261 Sep 11 20:14:19.822825 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:19.822796 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-16cd58ce-6b68959c7c-2rl4l"] Sep 11 20:14:19.823915 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:14:19.823893 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod145bf30f_2722_4c68_ab55_a9d05d03f80b.slice/crio-e70c0ed677f859ec4a7ae654a6c12582885f8df0b7632bed27e443976551c4bc WatchSource:0}: Error finding container e70c0ed677f859ec4a7ae654a6c12582885f8df0b7632bed27e443976551c4bc: Status 404 returned error can't find the container with id e70c0ed677f859ec4a7ae654a6c12582885f8df0b7632bed27e443976551c4bc Sep 11 20:14:20.200381 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:20.200289 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:20.455002 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:20.454908 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7151a8f6-a052-423a-a5be-aec5eec623d0" path="/var/lib/kubelet/pods/7151a8f6-a052-423a-a5be-aec5eec623d0/volumes" Sep 11 20:14:20.566193 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:20.566155 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-16cd58ce-6b68959c7c-2rl4l" event={"ID":"145bf30f-2722-4c68-ab55-a9d05d03f80b","Type":"ContainerStarted","Data":"3b3e22497199336593019e329542df53c76dae2c93a291c3c4ed65f2012e8fda"} Sep 11 20:14:20.566394 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:20.566217 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-16cd58ce-6b68959c7c-2rl4l" event={"ID":"145bf30f-2722-4c68-ab55-a9d05d03f80b","Type":"ContainerStarted","Data":"e70c0ed677f859ec4a7ae654a6c12582885f8df0b7632bed27e443976551c4bc"} Sep 11 20:14:20.566394 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:20.566249 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-16cd58ce-6b68959c7c-2rl4l" Sep 11 20:14:20.568042 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:20.568014 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-16cd58ce-7bc78c664b-zkwb2" event={"ID":"e382f502-335b-4dbe-9139-152a31cb28fc","Type":"ContainerStarted","Data":"c51125319533ed7597207b0c623bc9052c328c1ad34554e6ded074ca11afd546"} Sep 11 20:14:20.568143 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:20.568066 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-16cd58ce-7bc78c664b-zkwb2" event={"ID":"e382f502-335b-4dbe-9139-152a31cb28fc","Type":"ContainerStarted","Data":"c3aa5a60c732b03e6723c83742581a554bf4e9d9fda22a4e91a3547962d33261"} Sep 11 20:14:20.568143 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:20.568095 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-16cd58ce-7bc78c664b-zkwb2" Sep 11 20:14:20.586428 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:20.586387 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-16cd58ce-6b68959c7c-2rl4l" podStartSLOduration=1.58637477 podStartE2EDuration="1.58637477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:14:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:14:20.584472746 +0000 UTC m=+830.632523089" watchObservedRunningTime="2026-09-11 20:14:20.58637477 +0000 UTC m=+830.634425115" Sep 11 20:14:20.602075 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:20.602039 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-16cd58ce-7bc78c664b-zkwb2" podStartSLOduration=1.602025948 podStartE2EDuration="1.602025948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:14:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:14:20.600461506 +0000 UTC m=+830.648511875" watchObservedRunningTime="2026-09-11 20:14:20.602025948 +0000 UTC m=+830.650076291" Sep 11 20:14:20.945787 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:20.945755 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:21.049044 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:21.049008 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:21.725622 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:21.725566 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:31.575536 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:31.575502 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-16cd58ce-7bc78c664b-zkwb2" Sep 11 20:14:31.575998 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:31.575593 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-16cd58ce-6b68959c7c-2rl4l" Sep 11 20:14:39.032953 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:39.032918 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:40.592357 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:40.592322 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:41.421098 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:41.421060 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:42.311519 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:42.311464 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:43.241311 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:43.241274 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:44.050852 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:44.050818 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:44.667853 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:44.667824 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:44.826848 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:44.826818 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:45.639124 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:45.639092 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:46.286229 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:46.286199 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:47.036239 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:47.036209 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:47.712880 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:47.712841 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:47.748879 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:47.748847 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-16cd58ce-6b68959c7c-2rl4l"] Sep 11 20:14:47.749091 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:47.749049 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-16cd58ce-6b68959c7c-2rl4l" podUID="145bf30f-2722-4c68-ab55-a9d05d03f80b" containerName="payload-processing" containerID="cri-o://3b3e22497199336593019e329542df53c76dae2c93a291c3c4ed65f2012e8fda" gracePeriod=30 Sep 11 20:14:47.758421 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:47.758393 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-16cd58ce-7bc78c664b-zkwb2"] Sep 11 20:14:47.758678 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:47.758650 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-16cd58ce-7bc78c664b-zkwb2" podUID="e382f502-335b-4dbe-9139-152a31cb28fc" containerName="payload-pre-processing" containerID="cri-o://c51125319533ed7597207b0c623bc9052c328c1ad34554e6ded074ca11afd546" gracePeriod=30 Sep 11 20:14:47.789783 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:47.789755 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:47.850498 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:47.850469 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:47.915604 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:47.915543 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:47.980131 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:47.980082 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:48.029690 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:48.029660 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-16cd58ce-6b68959c7c-2rl4l" Sep 11 20:14:48.051116 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:48.051090 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-16cd58ce-7bc78c664b-zkwb2" Sep 11 20:14:48.199136 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:48.199103 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/145bf30f-2722-4c68-ab55-a9d05d03f80b-plugins-config-volume\") pod \"145bf30f-2722-4c68-ab55-a9d05d03f80b\" (UID: \"145bf30f-2722-4c68-ab55-a9d05d03f80b\") " Sep 11 20:14:48.199309 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:48.199194 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e382f502-335b-4dbe-9139-152a31cb28fc-plugins-config-volume\") pod \"e382f502-335b-4dbe-9139-152a31cb28fc\" (UID: \"e382f502-335b-4dbe-9139-152a31cb28fc\") " Sep 11 20:14:48.199309 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:48.199254 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxjdc\" (UniqueName: \"kubernetes.io/projected/145bf30f-2722-4c68-ab55-a9d05d03f80b-kube-api-access-sxjdc\") pod \"145bf30f-2722-4c68-ab55-a9d05d03f80b\" (UID: \"145bf30f-2722-4c68-ab55-a9d05d03f80b\") " Sep 11 20:14:48.199309 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:48.199300 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-78mh8\" (UniqueName: \"kubernetes.io/projected/e382f502-335b-4dbe-9139-152a31cb28fc-kube-api-access-78mh8\") pod \"e382f502-335b-4dbe-9139-152a31cb28fc\" (UID: \"e382f502-335b-4dbe-9139-152a31cb28fc\") " Sep 11 20:14:48.199516 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:48.199481 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/145bf30f-2722-4c68-ab55-a9d05d03f80b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "145bf30f-2722-4c68-ab55-a9d05d03f80b" (UID: "145bf30f-2722-4c68-ab55-a9d05d03f80b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:14:48.199650 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:48.199625 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e382f502-335b-4dbe-9139-152a31cb28fc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e382f502-335b-4dbe-9139-152a31cb28fc" (UID: "e382f502-335b-4dbe-9139-152a31cb28fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:14:48.199757 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:48.199741 2546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e382f502-335b-4dbe-9139-152a31cb28fc-plugins-config-volume\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:14:48.199807 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:48.199763 2546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/145bf30f-2722-4c68-ab55-a9d05d03f80b-plugins-config-volume\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:14:48.201235 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:48.201216 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e382f502-335b-4dbe-9139-152a31cb28fc-kube-api-access-78mh8" (OuterVolumeSpecName: "kube-api-access-78mh8") pod "e382f502-335b-4dbe-9139-152a31cb28fc" (UID: "e382f502-335b-4dbe-9139-152a31cb28fc"). InnerVolumeSpecName "kube-api-access-78mh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:14:48.201296 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:48.201257 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/145bf30f-2722-4c68-ab55-a9d05d03f80b-kube-api-access-sxjdc" (OuterVolumeSpecName: "kube-api-access-sxjdc") pod "145bf30f-2722-4c68-ab55-a9d05d03f80b" (UID: "145bf30f-2722-4c68-ab55-a9d05d03f80b"). InnerVolumeSpecName "kube-api-access-sxjdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:14:48.301007 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:48.300932 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78mh8\" (UniqueName: \"kubernetes.io/projected/e382f502-335b-4dbe-9139-152a31cb28fc-kube-api-access-78mh8\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:14:48.301007 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:48.300959 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxjdc\" (UniqueName: \"kubernetes.io/projected/145bf30f-2722-4c68-ab55-a9d05d03f80b-kube-api-access-sxjdc\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:14:48.402215 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:48.402179 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:48.724125 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:48.724092 2546 generic.go:356] "Generic (PLEG): container finished" podID="145bf30f-2722-4c68-ab55-a9d05d03f80b" containerID="3b3e22497199336593019e329542df53c76dae2c93a291c3c4ed65f2012e8fda" exitCode=0 Sep 11 20:14:48.724329 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:48.724155 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-16cd58ce-6b68959c7c-2rl4l" Sep 11 20:14:48.724329 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:48.724176 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-16cd58ce-6b68959c7c-2rl4l" event={"ID":"145bf30f-2722-4c68-ab55-a9d05d03f80b","Type":"ContainerDied","Data":"3b3e22497199336593019e329542df53c76dae2c93a291c3c4ed65f2012e8fda"} Sep 11 20:14:48.724329 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:48.724226 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-16cd58ce-6b68959c7c-2rl4l" event={"ID":"145bf30f-2722-4c68-ab55-a9d05d03f80b","Type":"ContainerDied","Data":"e70c0ed677f859ec4a7ae654a6c12582885f8df0b7632bed27e443976551c4bc"} Sep 11 20:14:48.724329 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:48.724255 2546 scope.go:117] "RemoveContainer" containerID="3b3e22497199336593019e329542df53c76dae2c93a291c3c4ed65f2012e8fda" Sep 11 20:14:48.725653 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:48.725611 2546 generic.go:356] "Generic (PLEG): container finished" podID="e382f502-335b-4dbe-9139-152a31cb28fc" containerID="c51125319533ed7597207b0c623bc9052c328c1ad34554e6ded074ca11afd546" exitCode=0 Sep 11 20:14:48.725753 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:48.725730 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-16cd58ce-7bc78c664b-zkwb2" event={"ID":"e382f502-335b-4dbe-9139-152a31cb28fc","Type":"ContainerDied","Data":"c51125319533ed7597207b0c623bc9052c328c1ad34554e6ded074ca11afd546"} Sep 11 20:14:48.725817 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:48.725777 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-16cd58ce-7bc78c664b-zkwb2" event={"ID":"e382f502-335b-4dbe-9139-152a31cb28fc","Type":"ContainerDied","Data":"c3aa5a60c732b03e6723c83742581a554bf4e9d9fda22a4e91a3547962d33261"} Sep 11 20:14:48.725817 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:48.725795 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-16cd58ce-7bc78c664b-zkwb2" Sep 11 20:14:48.734329 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:48.734310 2546 scope.go:117] "RemoveContainer" containerID="3b3e22497199336593019e329542df53c76dae2c93a291c3c4ed65f2012e8fda" Sep 11 20:14:48.734681 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:14:48.734661 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b3e22497199336593019e329542df53c76dae2c93a291c3c4ed65f2012e8fda\": container with ID starting with 3b3e22497199336593019e329542df53c76dae2c93a291c3c4ed65f2012e8fda not found: ID does not exist" containerID="3b3e22497199336593019e329542df53c76dae2c93a291c3c4ed65f2012e8fda" Sep 11 20:14:48.734777 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:48.734691 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b3e22497199336593019e329542df53c76dae2c93a291c3c4ed65f2012e8fda"} err="failed to get container status \"3b3e22497199336593019e329542df53c76dae2c93a291c3c4ed65f2012e8fda\": rpc error: code = NotFound desc = could not find container \"3b3e22497199336593019e329542df53c76dae2c93a291c3c4ed65f2012e8fda\": container with ID starting with 3b3e22497199336593019e329542df53c76dae2c93a291c3c4ed65f2012e8fda not found: ID does not exist" Sep 11 20:14:48.734777 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:48.734714 2546 scope.go:117] "RemoveContainer" containerID="c51125319533ed7597207b0c623bc9052c328c1ad34554e6ded074ca11afd546" Sep 11 20:14:48.742663 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:48.742643 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-16cd58ce-7bc78c664b-zkwb2"] Sep 11 20:14:48.743110 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:48.743090 2546 scope.go:117] "RemoveContainer" containerID="c51125319533ed7597207b0c623bc9052c328c1ad34554e6ded074ca11afd546" Sep 11 20:14:48.743382 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:14:48.743364 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c51125319533ed7597207b0c623bc9052c328c1ad34554e6ded074ca11afd546\": container with ID starting with c51125319533ed7597207b0c623bc9052c328c1ad34554e6ded074ca11afd546 not found: ID does not exist" containerID="c51125319533ed7597207b0c623bc9052c328c1ad34554e6ded074ca11afd546" Sep 11 20:14:48.743452 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:48.743387 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c51125319533ed7597207b0c623bc9052c328c1ad34554e6ded074ca11afd546"} err="failed to get container status \"c51125319533ed7597207b0c623bc9052c328c1ad34554e6ded074ca11afd546\": rpc error: code = NotFound desc = could not find container \"c51125319533ed7597207b0c623bc9052c328c1ad34554e6ded074ca11afd546\": container with ID starting with c51125319533ed7597207b0c623bc9052c328c1ad34554e6ded074ca11afd546 not found: ID does not exist" Sep 11 20:14:48.746371 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:48.746351 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-16cd58ce-7bc78c664b-zkwb2"] Sep 11 20:14:48.756079 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:48.756060 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-16cd58ce-6b68959c7c-2rl4l"] Sep 11 20:14:48.765124 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:48.765101 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-16cd58ce-6b68959c7c-2rl4l"] Sep 11 20:14:49.003208 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:49.003111 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:49.152074 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:49.152038 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:49.404682 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:49.404642 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:49.677334 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:49.676756 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795"] Sep 11 20:14:49.677489 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:49.677444 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e382f502-335b-4dbe-9139-152a31cb28fc" containerName="payload-pre-processing" Sep 11 20:14:49.677489 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:49.677462 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="e382f502-335b-4dbe-9139-152a31cb28fc" containerName="payload-pre-processing" Sep 11 20:14:49.677617 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:49.677499 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="145bf30f-2722-4c68-ab55-a9d05d03f80b" containerName="payload-processing" Sep 11 20:14:49.677617 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:49.677509 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="145bf30f-2722-4c68-ab55-a9d05d03f80b" containerName="payload-processing" Sep 11 20:14:49.677712 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:49.677664 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="145bf30f-2722-4c68-ab55-a9d05d03f80b" containerName="payload-processing" Sep 11 20:14:49.677712 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:49.677683 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="e382f502-335b-4dbe-9139-152a31cb28fc" containerName="payload-pre-processing" Sep 11 20:14:49.689614 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:49.683077 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795" Sep 11 20:14:49.689614 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:49.687438 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-f5b4f720-openshift-default-dockercfg-mnfb9\"" Sep 11 20:14:49.697945 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:49.695173 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795"] Sep 11 20:14:49.700144 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:49.700064 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:49.815234 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:49.815201 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eb3eda5-3f3d-4345-87f1-b24b1fd719bb-istio-token\") pod \"e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795\" (UID: \"3eb3eda5-3f3d-4345-87f1-b24b1fd719bb\") " pod="openshift-ingress/e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795" Sep 11 20:14:49.815234 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:49.815237 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eb3eda5-3f3d-4345-87f1-b24b1fd719bb-istio-envoy\") pod \"e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795\" (UID: \"3eb3eda5-3f3d-4345-87f1-b24b1fd719bb\") " pod="openshift-ingress/e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795" Sep 11 20:14:49.815458 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:49.815270 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eb3eda5-3f3d-4345-87f1-b24b1fd719bb-istio-podinfo\") pod \"e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795\" (UID: \"3eb3eda5-3f3d-4345-87f1-b24b1fd719bb\") " pod="openshift-ingress/e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795" Sep 11 20:14:49.815458 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:49.815319 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb3eda5-3f3d-4345-87f1-b24b1fd719bb-credential-socket\") pod \"e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795\" (UID: \"3eb3eda5-3f3d-4345-87f1-b24b1fd719bb\") " pod="openshift-ingress/e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795" Sep 11 20:14:49.815458 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:49.815346 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eb3eda5-3f3d-4345-87f1-b24b1fd719bb-istiod-ca-cert\") pod \"e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795\" (UID: \"3eb3eda5-3f3d-4345-87f1-b24b1fd719bb\") " pod="openshift-ingress/e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795" Sep 11 20:14:49.815458 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:49.815367 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eb3eda5-3f3d-4345-87f1-b24b1fd719bb-istio-data\") pod \"e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795\" (UID: \"3eb3eda5-3f3d-4345-87f1-b24b1fd719bb\") " pod="openshift-ingress/e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795" Sep 11 20:14:49.815458 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:49.815405 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb3eda5-3f3d-4345-87f1-b24b1fd719bb-workload-socket\") pod \"e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795\" (UID: \"3eb3eda5-3f3d-4345-87f1-b24b1fd719bb\") " pod="openshift-ingress/e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795" Sep 11 20:14:49.815458 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:49.815438 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4cs7\" (UniqueName: \"kubernetes.io/projected/3eb3eda5-3f3d-4345-87f1-b24b1fd719bb-kube-api-access-k4cs7\") pod \"e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795\" (UID: \"3eb3eda5-3f3d-4345-87f1-b24b1fd719bb\") " pod="openshift-ingress/e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795" Sep 11 20:14:49.815730 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:49.815534 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eb3eda5-3f3d-4345-87f1-b24b1fd719bb-workload-certs\") pod \"e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795\" (UID: \"3eb3eda5-3f3d-4345-87f1-b24b1fd719bb\") " pod="openshift-ingress/e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795" Sep 11 20:14:49.916319 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:49.916282 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eb3eda5-3f3d-4345-87f1-b24b1fd719bb-istio-envoy\") pod \"e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795\" (UID: \"3eb3eda5-3f3d-4345-87f1-b24b1fd719bb\") " pod="openshift-ingress/e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795" Sep 11 20:14:49.916496 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:49.916330 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eb3eda5-3f3d-4345-87f1-b24b1fd719bb-istio-podinfo\") pod \"e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795\" (UID: \"3eb3eda5-3f3d-4345-87f1-b24b1fd719bb\") " pod="openshift-ingress/e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795" Sep 11 20:14:49.916496 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:49.916362 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb3eda5-3f3d-4345-87f1-b24b1fd719bb-credential-socket\") pod \"e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795\" (UID: \"3eb3eda5-3f3d-4345-87f1-b24b1fd719bb\") " pod="openshift-ingress/e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795" Sep 11 20:14:49.916496 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:49.916387 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eb3eda5-3f3d-4345-87f1-b24b1fd719bb-istiod-ca-cert\") pod \"e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795\" (UID: \"3eb3eda5-3f3d-4345-87f1-b24b1fd719bb\") " pod="openshift-ingress/e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795" Sep 11 20:14:49.916496 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:49.916413 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eb3eda5-3f3d-4345-87f1-b24b1fd719bb-istio-data\") pod \"e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795\" (UID: \"3eb3eda5-3f3d-4345-87f1-b24b1fd719bb\") " pod="openshift-ingress/e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795" Sep 11 20:14:49.916790 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:49.916597 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb3eda5-3f3d-4345-87f1-b24b1fd719bb-workload-socket\") pod \"e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795\" (UID: \"3eb3eda5-3f3d-4345-87f1-b24b1fd719bb\") " pod="openshift-ingress/e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795" Sep 11 20:14:49.916790 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:49.916672 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4cs7\" (UniqueName: \"kubernetes.io/projected/3eb3eda5-3f3d-4345-87f1-b24b1fd719bb-kube-api-access-k4cs7\") pod \"e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795\" (UID: \"3eb3eda5-3f3d-4345-87f1-b24b1fd719bb\") " pod="openshift-ingress/e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795" Sep 11 20:14:49.916790 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:49.916746 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eb3eda5-3f3d-4345-87f1-b24b1fd719bb-workload-certs\") pod \"e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795\" (UID: \"3eb3eda5-3f3d-4345-87f1-b24b1fd719bb\") " pod="openshift-ingress/e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795" Sep 11 20:14:49.916938 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:49.916836 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eb3eda5-3f3d-4345-87f1-b24b1fd719bb-istio-token\") pod \"e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795\" (UID: \"3eb3eda5-3f3d-4345-87f1-b24b1fd719bb\") " pod="openshift-ingress/e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795" Sep 11 20:14:49.916938 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:49.916859 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eb3eda5-3f3d-4345-87f1-b24b1fd719bb-istio-data\") pod \"e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795\" (UID: \"3eb3eda5-3f3d-4345-87f1-b24b1fd719bb\") " pod="openshift-ingress/e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795" Sep 11 20:14:49.916938 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:49.916835 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb3eda5-3f3d-4345-87f1-b24b1fd719bb-credential-socket\") pod \"e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795\" (UID: \"3eb3eda5-3f3d-4345-87f1-b24b1fd719bb\") " pod="openshift-ingress/e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795" Sep 11 20:14:49.917105 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:49.917032 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb3eda5-3f3d-4345-87f1-b24b1fd719bb-workload-socket\") pod \"e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795\" (UID: \"3eb3eda5-3f3d-4345-87f1-b24b1fd719bb\") " pod="openshift-ingress/e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795" Sep 11 20:14:49.917105 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:49.917073 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eb3eda5-3f3d-4345-87f1-b24b1fd719bb-workload-certs\") pod \"e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795\" (UID: \"3eb3eda5-3f3d-4345-87f1-b24b1fd719bb\") " pod="openshift-ingress/e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795" Sep 11 20:14:49.917224 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:49.917201 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eb3eda5-3f3d-4345-87f1-b24b1fd719bb-istiod-ca-cert\") pod \"e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795\" (UID: \"3eb3eda5-3f3d-4345-87f1-b24b1fd719bb\") " pod="openshift-ingress/e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795" Sep 11 20:14:49.918718 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:49.918698 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eb3eda5-3f3d-4345-87f1-b24b1fd719bb-istio-envoy\") pod \"e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795\" (UID: \"3eb3eda5-3f3d-4345-87f1-b24b1fd719bb\") " pod="openshift-ingress/e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795" Sep 11 20:14:49.918853 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:49.918837 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eb3eda5-3f3d-4345-87f1-b24b1fd719bb-istio-podinfo\") pod \"e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795\" (UID: \"3eb3eda5-3f3d-4345-87f1-b24b1fd719bb\") " pod="openshift-ingress/e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795" Sep 11 20:14:49.945171 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:49.945106 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eb3eda5-3f3d-4345-87f1-b24b1fd719bb-istio-token\") pod \"e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795\" (UID: \"3eb3eda5-3f3d-4345-87f1-b24b1fd719bb\") " pod="openshift-ingress/e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795" Sep 11 20:14:49.945299 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:49.945249 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4cs7\" (UniqueName: \"kubernetes.io/projected/3eb3eda5-3f3d-4345-87f1-b24b1fd719bb-kube-api-access-k4cs7\") pod \"e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795\" (UID: \"3eb3eda5-3f3d-4345-87f1-b24b1fd719bb\") " pod="openshift-ingress/e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795" Sep 11 20:14:49.998946 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:49.998920 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795" Sep 11 20:14:50.091867 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:50.091841 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:50.192425 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:50.192391 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:50.275916 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:50.275890 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795"] Sep 11 20:14:50.277414 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:14:50.277387 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eb3eda5_3f3d_4345_87f1_b24b1fd719bb.slice/crio-d871dfd32b4b3f00463589dc82d3ad404045eed45bcd88b3bd641eb83c9b8086 WatchSource:0}: Error finding container d871dfd32b4b3f00463589dc82d3ad404045eed45bcd88b3bd641eb83c9b8086: Status 404 returned error can't find the container with id d871dfd32b4b3f00463589dc82d3ad404045eed45bcd88b3bd641eb83c9b8086 Sep 11 20:14:50.279647 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:50.279614 2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 20:14:50.279749 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:50.279690 2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 20:14:50.279749 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:50.279719 2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 20:14:50.451240 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:50.451196 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="145bf30f-2722-4c68-ab55-a9d05d03f80b" path="/var/lib/kubelet/pods/145bf30f-2722-4c68-ab55-a9d05d03f80b/volumes" Sep 11 20:14:50.451683 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:50.451661 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e382f502-335b-4dbe-9139-152a31cb28fc" path="/var/lib/kubelet/pods/e382f502-335b-4dbe-9139-152a31cb28fc/volumes" Sep 11 20:14:50.741112 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:50.741074 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795" event={"ID":"3eb3eda5-3f3d-4345-87f1-b24b1fd719bb","Type":"ContainerStarted","Data":"a56018aa4635859aa05bea293ab1a40edf5e86b901097789cf63f091dca4b35c"} Sep 11 20:14:50.741293 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:50.741132 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795" event={"ID":"3eb3eda5-3f3d-4345-87f1-b24b1fd719bb","Type":"ContainerStarted","Data":"d871dfd32b4b3f00463589dc82d3ad404045eed45bcd88b3bd641eb83c9b8086"} Sep 11 20:14:50.762331 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:50.762273 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795" podStartSLOduration=1.762252717 podStartE2EDuration="1.762252717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:14:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:14:50.758230188 +0000 UTC m=+860.806280533" watchObservedRunningTime="2026-09-11 20:14:50.762252717 +0000 UTC m=+860.810303061" Sep 11 20:14:50.907756 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:50.907718 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:51.000043 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:50.999957 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795" Sep 11 20:14:51.727103 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:51.727065 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:52.005382 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:52.005304 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795" Sep 11 20:14:52.156675 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:52.156626 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:52.288360 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:52.288280 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:52.735296 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:52.735263 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:52.752930 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:52.752907 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795" Sep 11 20:14:52.753924 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:52.753903 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795" Sep 11 20:14:53.446267 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:53.446237 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:53.549463 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:53.549426 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:53.837191 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:53.837106 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:54.027588 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:54.027553 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:54.134910 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:54.134830 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:54.138175 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:54.138149 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f5b4f720-f4dc66c8-5f8nq"] Sep 11 20:14:54.143114 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:54.143094 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-f5b4f720-f4dc66c8-5f8nq" Sep 11 20:14:54.145145 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:54.145123 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-f5b4f720\"" Sep 11 20:14:54.145249 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:54.145134 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-f5b4f720-dockercfg-twpdb\"" Sep 11 20:14:54.153039 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:54.153019 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f5b4f720-f4dc66c8-5f8nq"] Sep 11 20:14:54.153255 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:54.153233 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxdv2\" (UniqueName: \"kubernetes.io/projected/6699d0e0-fd61-4739-82a2-30e1b96e2411-kube-api-access-kxdv2\") pod \"payload-pre-processing-e2e-derive-f5b4f720-f4dc66c8-5f8nq\" (UID: \"6699d0e0-fd61-4739-82a2-30e1b96e2411\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f5b4f720-f4dc66c8-5f8nq" Sep 11 20:14:54.153327 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:54.153286 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6699d0e0-fd61-4739-82a2-30e1b96e2411-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-f5b4f720-f4dc66c8-5f8nq\" (UID: \"6699d0e0-fd61-4739-82a2-30e1b96e2411\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f5b4f720-f4dc66c8-5f8nq" Sep 11 20:14:54.160260 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:54.160240 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f5b4f720-6959cb4fbf-cjcsq"] Sep 11 20:14:54.164101 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:54.164083 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-f5b4f720-6959cb4fbf-cjcsq" Sep 11 20:14:54.174941 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:54.174922 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f5b4f720-6959cb4fbf-cjcsq"] Sep 11 20:14:54.254531 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:54.254471 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkrzp\" (UniqueName: \"kubernetes.io/projected/51699108-a2b1-4bdb-bede-c7020defe411-kube-api-access-fkrzp\") pod \"payload-processing-e2e-derive-f5b4f720-6959cb4fbf-cjcsq\" (UID: \"51699108-a2b1-4bdb-bede-c7020defe411\") " pod="openshift-ingress/payload-processing-e2e-derive-f5b4f720-6959cb4fbf-cjcsq" Sep 11 20:14:54.254531 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:54.254533 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51699108-a2b1-4bdb-bede-c7020defe411-plugins-config-volume\") pod \"payload-processing-e2e-derive-f5b4f720-6959cb4fbf-cjcsq\" (UID: \"51699108-a2b1-4bdb-bede-c7020defe411\") " pod="openshift-ingress/payload-processing-e2e-derive-f5b4f720-6959cb4fbf-cjcsq" Sep 11 20:14:54.254894 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:54.254735 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxdv2\" (UniqueName: \"kubernetes.io/projected/6699d0e0-fd61-4739-82a2-30e1b96e2411-kube-api-access-kxdv2\") pod \"payload-pre-processing-e2e-derive-f5b4f720-f4dc66c8-5f8nq\" (UID: \"6699d0e0-fd61-4739-82a2-30e1b96e2411\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f5b4f720-f4dc66c8-5f8nq" Sep 11 20:14:54.254894 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:54.254787 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6699d0e0-fd61-4739-82a2-30e1b96e2411-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-f5b4f720-f4dc66c8-5f8nq\" (UID: \"6699d0e0-fd61-4739-82a2-30e1b96e2411\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f5b4f720-f4dc66c8-5f8nq" Sep 11 20:14:54.255704 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:54.255623 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6699d0e0-fd61-4739-82a2-30e1b96e2411-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-f5b4f720-f4dc66c8-5f8nq\" (UID: \"6699d0e0-fd61-4739-82a2-30e1b96e2411\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f5b4f720-f4dc66c8-5f8nq" Sep 11 20:14:54.264555 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:54.264516 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxdv2\" (UniqueName: \"kubernetes.io/projected/6699d0e0-fd61-4739-82a2-30e1b96e2411-kube-api-access-kxdv2\") pod \"payload-pre-processing-e2e-derive-f5b4f720-f4dc66c8-5f8nq\" (UID: \"6699d0e0-fd61-4739-82a2-30e1b96e2411\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f5b4f720-f4dc66c8-5f8nq" Sep 11 20:14:54.356058 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:54.356027 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkrzp\" (UniqueName: \"kubernetes.io/projected/51699108-a2b1-4bdb-bede-c7020defe411-kube-api-access-fkrzp\") pod \"payload-processing-e2e-derive-f5b4f720-6959cb4fbf-cjcsq\" (UID: \"51699108-a2b1-4bdb-bede-c7020defe411\") " pod="openshift-ingress/payload-processing-e2e-derive-f5b4f720-6959cb4fbf-cjcsq" Sep 11 20:14:54.356231 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:54.356073 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51699108-a2b1-4bdb-bede-c7020defe411-plugins-config-volume\") pod \"payload-processing-e2e-derive-f5b4f720-6959cb4fbf-cjcsq\" (UID: \"51699108-a2b1-4bdb-bede-c7020defe411\") " pod="openshift-ingress/payload-processing-e2e-derive-f5b4f720-6959cb4fbf-cjcsq" Sep 11 20:14:54.356801 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:54.356776 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51699108-a2b1-4bdb-bede-c7020defe411-plugins-config-volume\") pod \"payload-processing-e2e-derive-f5b4f720-6959cb4fbf-cjcsq\" (UID: \"51699108-a2b1-4bdb-bede-c7020defe411\") " pod="openshift-ingress/payload-processing-e2e-derive-f5b4f720-6959cb4fbf-cjcsq" Sep 11 20:14:54.366565 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:54.366543 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkrzp\" (UniqueName: \"kubernetes.io/projected/51699108-a2b1-4bdb-bede-c7020defe411-kube-api-access-fkrzp\") pod \"payload-processing-e2e-derive-f5b4f720-6959cb4fbf-cjcsq\" (UID: \"51699108-a2b1-4bdb-bede-c7020defe411\") " pod="openshift-ingress/payload-processing-e2e-derive-f5b4f720-6959cb4fbf-cjcsq" Sep 11 20:14:54.455721 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:54.455641 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-f5b4f720-f4dc66c8-5f8nq" Sep 11 20:14:54.476238 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:54.476207 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-f5b4f720-6959cb4fbf-cjcsq" Sep 11 20:14:54.555960 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:54.555915 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:54.734003 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:54.733979 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f5b4f720-f4dc66c8-5f8nq"] Sep 11 20:14:54.735688 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:14:54.735661 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6699d0e0_fd61_4739_82a2_30e1b96e2411.slice/crio-d423e459e6250f23576e122bb904193377ec6d0c9cc85068b57196b45f0554e0 WatchSource:0}: Error finding container d423e459e6250f23576e122bb904193377ec6d0c9cc85068b57196b45f0554e0: Status 404 returned error can't find the container with id d423e459e6250f23576e122bb904193377ec6d0c9cc85068b57196b45f0554e0 Sep 11 20:14:54.764758 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:54.764730 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-f5b4f720-f4dc66c8-5f8nq" event={"ID":"6699d0e0-fd61-4739-82a2-30e1b96e2411","Type":"ContainerStarted","Data":"d423e459e6250f23576e122bb904193377ec6d0c9cc85068b57196b45f0554e0"} Sep 11 20:14:54.769068 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:54.769045 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f5b4f720-6959cb4fbf-cjcsq"] Sep 11 20:14:54.770731 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:14:54.770704 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51699108_a2b1_4bdb_bede_c7020defe411.slice/crio-261c7968b4b7c567579960a5a73986d9fc0802f4b534bdb8b5f2cfbe356be49a WatchSource:0}: Error finding container 261c7968b4b7c567579960a5a73986d9fc0802f4b534bdb8b5f2cfbe356be49a: Status 404 returned error can't find the container with id 261c7968b4b7c567579960a5a73986d9fc0802f4b534bdb8b5f2cfbe356be49a Sep 11 20:14:55.335161 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:55.335126 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:55.771979 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:55.771946 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-f5b4f720-6959cb4fbf-cjcsq" event={"ID":"51699108-a2b1-4bdb-bede-c7020defe411","Type":"ContainerStarted","Data":"2501d7a094622418d9b9efd74d40140db1ddc9d8972c96d2a5b5665927512104"} Sep 11 20:14:55.772151 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:55.772043 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-f5b4f720-6959cb4fbf-cjcsq" event={"ID":"51699108-a2b1-4bdb-bede-c7020defe411","Type":"ContainerStarted","Data":"261c7968b4b7c567579960a5a73986d9fc0802f4b534bdb8b5f2cfbe356be49a"} Sep 11 20:14:55.772151 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:55.772077 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-f5b4f720-6959cb4fbf-cjcsq" Sep 11 20:14:55.773848 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:55.773824 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-f5b4f720-f4dc66c8-5f8nq" event={"ID":"6699d0e0-fd61-4739-82a2-30e1b96e2411","Type":"ContainerStarted","Data":"ce1a29fce66b59468341570493e4a2d111d4c9a0d3f4fca3cb6a65f20913b68c"} Sep 11 20:14:55.773989 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:55.773874 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-f5b4f720-f4dc66c8-5f8nq" Sep 11 20:14:55.791027 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:55.790981 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-f5b4f720-6959cb4fbf-cjcsq" podStartSLOduration=1.79096578 podStartE2EDuration="1.79096578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:14:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:14:55.788291956 +0000 UTC m=+865.836342301" watchObservedRunningTime="2026-09-11 20:14:55.79096578 +0000 UTC m=+865.839016118" Sep 11 20:14:55.811289 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:55.811247 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-f5b4f720-f4dc66c8-5f8nq" podStartSLOduration=1.811235664 podStartE2EDuration="1.811235664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:14:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:14:55.80884559 +0000 UTC m=+865.856895946" watchObservedRunningTime="2026-09-11 20:14:55.811235664 +0000 UTC m=+865.859286008" Sep 11 20:14:56.299703 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:56.299664 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:57.170695 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:57.170653 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:57.902633 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:14:57.902603 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:15:05.697258 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:05.697224 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:15:06.781017 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:06.780987 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-f5b4f720-6959cb4fbf-cjcsq" Sep 11 20:15:06.781377 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:06.781161 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-f5b4f720-f4dc66c8-5f8nq" Sep 11 20:15:12.275827 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:12.275778 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:15:12.377358 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:12.377306 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:15:15.641116 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:15.641078 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:15:16.610721 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:16.610689 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:15:16.906677 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:16.906565 2546 generic.go:356] "Generic (PLEG): container finished" podID="3eb3eda5-3f3d-4345-87f1-b24b1fd719bb" containerID="a56018aa4635859aa05bea293ab1a40edf5e86b901097789cf63f091dca4b35c" exitCode=0 Sep 11 20:15:16.906677 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:16.906642 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795" event={"ID":"3eb3eda5-3f3d-4345-87f1-b24b1fd719bb","Type":"ContainerDied","Data":"a56018aa4635859aa05bea293ab1a40edf5e86b901097789cf63f091dca4b35c"} Sep 11 20:15:16.907161 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:16.907133 2546 scope.go:117] "RemoveContainer" containerID="a56018aa4635859aa05bea293ab1a40edf5e86b901097789cf63f091dca4b35c" Sep 11 20:15:16.908448 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:16.908420 2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 20:15:16.908515 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:16.908489 2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 20:15:16.908552 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:16.908522 2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 20:15:17.533019 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:17.532970 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:15:17.914287 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:17.914257 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795" event={"ID":"3eb3eda5-3f3d-4345-87f1-b24b1fd719bb","Type":"ContainerStarted","Data":"9fcc169600bf41619f3f0ba85397d170e17fb5e16ce156ba0c4c9e276425b6a6"} Sep 11 20:15:17.999828 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:17.999792 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795" Sep 11 20:15:18.000901 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:18.000875 2546 patch_prober.go:28] interesting pod/e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Sep 11 20:15:18.000998 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:18.000934 2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795" podUID="3eb3eda5-3f3d-4345-87f1-b24b1fd719bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Sep 11 20:15:18.332152 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:18.332043 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:15:18.999824 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:18.999778 2546 patch_prober.go:28] interesting pod/e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Sep 11 20:15:19.000305 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:18.999847 2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795" podUID="3eb3eda5-3f3d-4345-87f1-b24b1fd719bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Sep 11 20:15:19.214064 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:19.214032 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:15:19.999461 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:19.999425 2546 patch_prober.go:28] interesting pod/e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Sep 11 20:15:19.999658 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:19.999480 2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795" podUID="3eb3eda5-3f3d-4345-87f1-b24b1fd719bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Sep 11 20:15:20.220321 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:20.220284 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:15:20.999504 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:20.999463 2546 patch_prober.go:28] interesting pod/e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Sep 11 20:15:20.999697 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:20.999520 2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795" podUID="3eb3eda5-3f3d-4345-87f1-b24b1fd719bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Sep 11 20:15:21.133753 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:21.133710 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:15:21.942048 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:21.942005 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:15:21.999483 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:21.999447 2546 patch_prober.go:28] interesting pod/e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Sep 11 20:15:21.999713 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:21.999508 2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795" podUID="3eb3eda5-3f3d-4345-87f1-b24b1fd719bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Sep 11 20:15:22.753233 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:22.753199 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795" Sep 11 20:15:22.796553 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:22.796520 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:15:23.004726 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:23.004653 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795" Sep 11 20:15:23.006011 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:23.005988 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795" Sep 11 20:15:24.112076 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:24.112039 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:15:24.215570 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:24.215525 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:15:24.908541 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:24.908490 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:15:25.710204 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:25.710167 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:15:26.298773 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:26.298736 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:15:26.319487 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:26.319455 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f5b4f720-6959cb4fbf-cjcsq"] Sep 11 20:15:26.319717 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:26.319685 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-f5b4f720-6959cb4fbf-cjcsq" podUID="51699108-a2b1-4bdb-bede-c7020defe411" containerName="payload-processing" containerID="cri-o://2501d7a094622418d9b9efd74d40140db1ddc9d8972c96d2a5b5665927512104" gracePeriod=30 Sep 11 20:15:26.332418 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:26.332389 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f5b4f720-f4dc66c8-5f8nq"] Sep 11 20:15:26.332909 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:26.332879 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-f5b4f720-f4dc66c8-5f8nq" podUID="6699d0e0-fd61-4739-82a2-30e1b96e2411" containerName="payload-pre-processing" containerID="cri-o://ce1a29fce66b59468341570493e4a2d111d4c9a0d3f4fca3cb6a65f20913b68c" gracePeriod=30 Sep 11 20:15:26.398778 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:26.398745 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:15:26.511454 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:26.511359 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:15:26.619988 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:26.619965 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-f5b4f720-f4dc66c8-5f8nq" Sep 11 20:15:26.622886 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:26.622868 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-f5b4f720-6959cb4fbf-cjcsq" Sep 11 20:15:26.634133 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:26.634103 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:15:26.691712 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:26.691683 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795"] Sep 11 20:15:26.691891 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:26.691868 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795" podUID="3eb3eda5-3f3d-4345-87f1-b24b1fd719bb" containerName="istio-proxy" containerID="cri-o://9fcc169600bf41619f3f0ba85397d170e17fb5e16ce156ba0c4c9e276425b6a6" gracePeriod=30 Sep 11 20:15:26.716631 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:26.716596 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fkrzp\" (UniqueName: \"kubernetes.io/projected/51699108-a2b1-4bdb-bede-c7020defe411-kube-api-access-fkrzp\") pod \"51699108-a2b1-4bdb-bede-c7020defe411\" (UID: \"51699108-a2b1-4bdb-bede-c7020defe411\") " Sep 11 20:15:26.717072 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:26.716772 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kxdv2\" (UniqueName: \"kubernetes.io/projected/6699d0e0-fd61-4739-82a2-30e1b96e2411-kube-api-access-kxdv2\") pod \"6699d0e0-fd61-4739-82a2-30e1b96e2411\" (UID: \"6699d0e0-fd61-4739-82a2-30e1b96e2411\") " Sep 11 20:15:26.717072 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:26.716858 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51699108-a2b1-4bdb-bede-c7020defe411-plugins-config-volume\") pod \"51699108-a2b1-4bdb-bede-c7020defe411\" (UID: \"51699108-a2b1-4bdb-bede-c7020defe411\") " Sep 11 20:15:26.717072 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:26.716899 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6699d0e0-fd61-4739-82a2-30e1b96e2411-plugins-config-volume\") pod \"6699d0e0-fd61-4739-82a2-30e1b96e2411\" (UID: \"6699d0e0-fd61-4739-82a2-30e1b96e2411\") " Sep 11 20:15:26.717257 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:26.717235 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51699108-a2b1-4bdb-bede-c7020defe411-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "51699108-a2b1-4bdb-bede-c7020defe411" (UID: "51699108-a2b1-4bdb-bede-c7020defe411"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:15:26.717348 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:26.717310 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6699d0e0-fd61-4739-82a2-30e1b96e2411-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6699d0e0-fd61-4739-82a2-30e1b96e2411" (UID: "6699d0e0-fd61-4739-82a2-30e1b96e2411"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:15:26.717400 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:26.717387 2546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51699108-a2b1-4bdb-bede-c7020defe411-plugins-config-volume\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:15:26.719035 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:26.719000 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6699d0e0-fd61-4739-82a2-30e1b96e2411-kube-api-access-kxdv2" (OuterVolumeSpecName: "kube-api-access-kxdv2") pod "6699d0e0-fd61-4739-82a2-30e1b96e2411" (UID: "6699d0e0-fd61-4739-82a2-30e1b96e2411"). InnerVolumeSpecName "kube-api-access-kxdv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:15:26.719144 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:26.719073 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51699108-a2b1-4bdb-bede-c7020defe411-kube-api-access-fkrzp" (OuterVolumeSpecName: "kube-api-access-fkrzp") pod "51699108-a2b1-4bdb-bede-c7020defe411" (UID: "51699108-a2b1-4bdb-bede-c7020defe411"). InnerVolumeSpecName "kube-api-access-fkrzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:15:26.722198 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:26.722175 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:15:26.818017 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:26.817980 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxdv2\" (UniqueName: \"kubernetes.io/projected/6699d0e0-fd61-4739-82a2-30e1b96e2411-kube-api-access-kxdv2\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:15:26.818017 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:26.818008 2546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6699d0e0-fd61-4739-82a2-30e1b96e2411-plugins-config-volume\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:15:26.818017 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:26.818019 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkrzp\" (UniqueName: \"kubernetes.io/projected/51699108-a2b1-4bdb-bede-c7020defe411-kube-api-access-fkrzp\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:15:26.944930 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:26.944844 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:15:26.984034 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:26.983997 2546 generic.go:356] "Generic (PLEG): container finished" podID="51699108-a2b1-4bdb-bede-c7020defe411" containerID="2501d7a094622418d9b9efd74d40140db1ddc9d8972c96d2a5b5665927512104" exitCode=0 Sep 11 20:15:26.984178 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:26.984048 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-f5b4f720-6959cb4fbf-cjcsq" Sep 11 20:15:26.984178 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:26.984087 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-f5b4f720-6959cb4fbf-cjcsq" event={"ID":"51699108-a2b1-4bdb-bede-c7020defe411","Type":"ContainerDied","Data":"2501d7a094622418d9b9efd74d40140db1ddc9d8972c96d2a5b5665927512104"} Sep 11 20:15:26.984178 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:26.984145 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-f5b4f720-6959cb4fbf-cjcsq" event={"ID":"51699108-a2b1-4bdb-bede-c7020defe411","Type":"ContainerDied","Data":"261c7968b4b7c567579960a5a73986d9fc0802f4b534bdb8b5f2cfbe356be49a"} Sep 11 20:15:26.984178 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:26.984168 2546 scope.go:117] "RemoveContainer" containerID="2501d7a094622418d9b9efd74d40140db1ddc9d8972c96d2a5b5665927512104" Sep 11 20:15:26.985610 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:26.985562 2546 generic.go:356] "Generic (PLEG): container finished" podID="6699d0e0-fd61-4739-82a2-30e1b96e2411" containerID="ce1a29fce66b59468341570493e4a2d111d4c9a0d3f4fca3cb6a65f20913b68c" exitCode=0 Sep 11 20:15:26.985744 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:26.985614 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-f5b4f720-f4dc66c8-5f8nq" event={"ID":"6699d0e0-fd61-4739-82a2-30e1b96e2411","Type":"ContainerDied","Data":"ce1a29fce66b59468341570493e4a2d111d4c9a0d3f4fca3cb6a65f20913b68c"} Sep 11 20:15:26.985744 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:26.985660 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-f5b4f720-f4dc66c8-5f8nq" event={"ID":"6699d0e0-fd61-4739-82a2-30e1b96e2411","Type":"ContainerDied","Data":"d423e459e6250f23576e122bb904193377ec6d0c9cc85068b57196b45f0554e0"} Sep 11 20:15:26.985744 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:26.985622 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-f5b4f720-f4dc66c8-5f8nq" Sep 11 20:15:26.994790 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:26.994765 2546 scope.go:117] "RemoveContainer" containerID="2501d7a094622418d9b9efd74d40140db1ddc9d8972c96d2a5b5665927512104" Sep 11 20:15:26.995370 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:15:26.995224 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2501d7a094622418d9b9efd74d40140db1ddc9d8972c96d2a5b5665927512104\": container with ID starting with 2501d7a094622418d9b9efd74d40140db1ddc9d8972c96d2a5b5665927512104 not found: ID does not exist" containerID="2501d7a094622418d9b9efd74d40140db1ddc9d8972c96d2a5b5665927512104" Sep 11 20:15:26.995370 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:26.995257 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2501d7a094622418d9b9efd74d40140db1ddc9d8972c96d2a5b5665927512104"} err="failed to get container status \"2501d7a094622418d9b9efd74d40140db1ddc9d8972c96d2a5b5665927512104\": rpc error: code = NotFound desc = could not find container \"2501d7a094622418d9b9efd74d40140db1ddc9d8972c96d2a5b5665927512104\": container with ID starting with 2501d7a094622418d9b9efd74d40140db1ddc9d8972c96d2a5b5665927512104 not found: ID does not exist" Sep 11 20:15:26.995370 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:26.995280 2546 scope.go:117] "RemoveContainer" containerID="ce1a29fce66b59468341570493e4a2d111d4c9a0d3f4fca3cb6a65f20913b68c" Sep 11 20:15:27.010873 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:27.010846 2546 scope.go:117] "RemoveContainer" containerID="ce1a29fce66b59468341570493e4a2d111d4c9a0d3f4fca3cb6a65f20913b68c" Sep 11 20:15:27.011251 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:15:27.011223 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce1a29fce66b59468341570493e4a2d111d4c9a0d3f4fca3cb6a65f20913b68c\": container with ID starting with ce1a29fce66b59468341570493e4a2d111d4c9a0d3f4fca3cb6a65f20913b68c not found: ID does not exist" containerID="ce1a29fce66b59468341570493e4a2d111d4c9a0d3f4fca3cb6a65f20913b68c" Sep 11 20:15:27.011338 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:27.011261 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce1a29fce66b59468341570493e4a2d111d4c9a0d3f4fca3cb6a65f20913b68c"} err="failed to get container status \"ce1a29fce66b59468341570493e4a2d111d4c9a0d3f4fca3cb6a65f20913b68c\": rpc error: code = NotFound desc = could not find container \"ce1a29fce66b59468341570493e4a2d111d4c9a0d3f4fca3cb6a65f20913b68c\": container with ID starting with ce1a29fce66b59468341570493e4a2d111d4c9a0d3f4fca3cb6a65f20913b68c not found: ID does not exist" Sep 11 20:15:27.014867 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:27.014821 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f5b4f720-f4dc66c8-5f8nq"] Sep 11 20:15:27.017795 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:27.017774 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f5b4f720-f4dc66c8-5f8nq"] Sep 11 20:15:27.028176 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:27.028146 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f5b4f720-6959cb4fbf-cjcsq"] Sep 11 20:15:27.031693 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:27.031672 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f5b4f720-6959cb4fbf-cjcsq"] Sep 11 20:15:27.353506 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:27.353465 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:15:27.500080 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:27.500042 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:15:28.027453 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:28.027413 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:15:28.451723 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:28.451695 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51699108-a2b1-4bdb-bede-c7020defe411" path="/var/lib/kubelet/pods/51699108-a2b1-4bdb-bede-c7020defe411/volumes" Sep 11 20:15:28.452099 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:28.452085 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6699d0e0-fd61-4739-82a2-30e1b96e2411" path="/var/lib/kubelet/pods/6699d0e0-fd61-4739-82a2-30e1b96e2411/volumes" Sep 11 20:15:28.747925 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:28.747807 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:15:28.846296 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:28.846262 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:15:29.546001 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:29.545950 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:15:30.117935 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:30.117900 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-46da77-8b9b66cc9-zzk5q"] Sep 11 20:15:30.118257 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:30.118205 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-46da77-8b9b66cc9-zzk5q" podUID="ed23046d-f478-42bd-a143-f0add41d8342" containerName="maas-api" containerID="cri-o://9fc453054bb0c07b7f4dab076db7a249d3c4926b359ab19f8dba5a0aa2b55d3c" gracePeriod=30 Sep 11 20:15:30.144583 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:30.144543 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:15:30.213167 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:30.213124 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:15:30.275218 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:30.275181 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:15:30.376558 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:30.376471 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:15:30.401962 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:30.401935 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-46da77-8b9b66cc9-zzk5q" Sep 11 20:15:30.456330 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:30.456301 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed23046d-f478-42bd-a143-f0add41d8342-openshift-service-ca\") pod \"ed23046d-f478-42bd-a143-f0add41d8342\" (UID: \"ed23046d-f478-42bd-a143-f0add41d8342\") " Sep 11 20:15:30.456497 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:30.456394 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed23046d-f478-42bd-a143-f0add41d8342-metrics-certs\") pod \"ed23046d-f478-42bd-a143-f0add41d8342\" (UID: \"ed23046d-f478-42bd-a143-f0add41d8342\") " Sep 11 20:15:30.456497 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:30.456463 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed23046d-f478-42bd-a143-f0add41d8342-maas-api-tls\") pod \"ed23046d-f478-42bd-a143-f0add41d8342\" (UID: \"ed23046d-f478-42bd-a143-f0add41d8342\") " Sep 11 20:15:30.456712 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:30.456507 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8tw7\" (UniqueName: \"kubernetes.io/projected/ed23046d-f478-42bd-a143-f0add41d8342-kube-api-access-n8tw7\") pod \"ed23046d-f478-42bd-a143-f0add41d8342\" (UID: \"ed23046d-f478-42bd-a143-f0add41d8342\") " Sep 11 20:15:30.456804 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:30.456775 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed23046d-f478-42bd-a143-f0add41d8342-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ed23046d-f478-42bd-a143-f0add41d8342" (UID: "ed23046d-f478-42bd-a143-f0add41d8342"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:15:30.456920 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:30.456893 2546 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed23046d-f478-42bd-a143-f0add41d8342-openshift-service-ca\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:15:30.459376 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:30.459271 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed23046d-f478-42bd-a143-f0add41d8342-kube-api-access-n8tw7" (OuterVolumeSpecName: "kube-api-access-n8tw7") pod "ed23046d-f478-42bd-a143-f0add41d8342" (UID: "ed23046d-f478-42bd-a143-f0add41d8342"). InnerVolumeSpecName "kube-api-access-n8tw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:15:30.459784 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:30.459750 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed23046d-f478-42bd-a143-f0add41d8342-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ed23046d-f478-42bd-a143-f0add41d8342" (UID: "ed23046d-f478-42bd-a143-f0add41d8342"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 20:15:30.459899 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:30.459784 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed23046d-f478-42bd-a143-f0add41d8342-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ed23046d-f478-42bd-a143-f0add41d8342" (UID: "ed23046d-f478-42bd-a143-f0add41d8342"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 20:15:30.481451 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:30.481427 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:15:30.557843 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:30.557808 2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed23046d-f478-42bd-a143-f0add41d8342-maas-api-tls\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:15:30.557843 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:30.557843 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8tw7\" (UniqueName: \"kubernetes.io/projected/ed23046d-f478-42bd-a143-f0add41d8342-kube-api-access-n8tw7\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:15:30.558296 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:30.557857 2546 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed23046d-f478-42bd-a143-f0add41d8342-metrics-certs\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:15:30.717635 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:30.717519 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:15:30.862174 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:30.862129 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:15:31.011921 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:31.011828 2546 generic.go:356] "Generic (PLEG): container finished" podID="ed23046d-f478-42bd-a143-f0add41d8342" containerID="9fc453054bb0c07b7f4dab076db7a249d3c4926b359ab19f8dba5a0aa2b55d3c" exitCode=0 Sep 11 20:15:31.011921 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:31.011890 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-46da77-8b9b66cc9-zzk5q" Sep 11 20:15:31.011921 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:31.011908 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-46da77-8b9b66cc9-zzk5q" event={"ID":"ed23046d-f478-42bd-a143-f0add41d8342","Type":"ContainerDied","Data":"9fc453054bb0c07b7f4dab076db7a249d3c4926b359ab19f8dba5a0aa2b55d3c"} Sep 11 20:15:31.012194 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:31.012001 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-46da77-8b9b66cc9-zzk5q" event={"ID":"ed23046d-f478-42bd-a143-f0add41d8342","Type":"ContainerDied","Data":"33286e6cc03341fd9d9e2ac5e209713f66816aaa21857641de1b2bf775423c83"} Sep 11 20:15:31.012194 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:31.012036 2546 scope.go:117] "RemoveContainer" containerID="9fc453054bb0c07b7f4dab076db7a249d3c4926b359ab19f8dba5a0aa2b55d3c" Sep 11 20:15:31.020678 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:31.020661 2546 scope.go:117] "RemoveContainer" containerID="9fc453054bb0c07b7f4dab076db7a249d3c4926b359ab19f8dba5a0aa2b55d3c" Sep 11 20:15:31.020968 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:15:31.020947 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fc453054bb0c07b7f4dab076db7a249d3c4926b359ab19f8dba5a0aa2b55d3c\": container with ID starting with 9fc453054bb0c07b7f4dab076db7a249d3c4926b359ab19f8dba5a0aa2b55d3c not found: ID does not exist" containerID="9fc453054bb0c07b7f4dab076db7a249d3c4926b359ab19f8dba5a0aa2b55d3c" Sep 11 20:15:31.021067 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:31.020973 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fc453054bb0c07b7f4dab076db7a249d3c4926b359ab19f8dba5a0aa2b55d3c"} err="failed to get container status \"9fc453054bb0c07b7f4dab076db7a249d3c4926b359ab19f8dba5a0aa2b55d3c\": rpc error: code = NotFound desc = could not find container \"9fc453054bb0c07b7f4dab076db7a249d3c4926b359ab19f8dba5a0aa2b55d3c\": container with ID starting with 9fc453054bb0c07b7f4dab076db7a249d3c4926b359ab19f8dba5a0aa2b55d3c not found: ID does not exist" Sep 11 20:15:31.032508 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:31.032488 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-46da77-8b9b66cc9-zzk5q"] Sep 11 20:15:31.036002 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:31.035981 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-46da77-8b9b66cc9-zzk5q"] Sep 11 20:15:31.618825 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:31.618792 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:15:31.956373 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:31.956350 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795" Sep 11 20:15:32.020259 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:32.020227 2546 generic.go:356] "Generic (PLEG): container finished" podID="3eb3eda5-3f3d-4345-87f1-b24b1fd719bb" containerID="9fcc169600bf41619f3f0ba85397d170e17fb5e16ce156ba0c4c9e276425b6a6" exitCode=0 Sep 11 20:15:32.020426 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:32.020272 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795" Sep 11 20:15:32.020426 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:32.020308 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795" event={"ID":"3eb3eda5-3f3d-4345-87f1-b24b1fd719bb","Type":"ContainerDied","Data":"9fcc169600bf41619f3f0ba85397d170e17fb5e16ce156ba0c4c9e276425b6a6"} Sep 11 20:15:32.020426 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:32.020362 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795" event={"ID":"3eb3eda5-3f3d-4345-87f1-b24b1fd719bb","Type":"ContainerDied","Data":"d871dfd32b4b3f00463589dc82d3ad404045eed45bcd88b3bd641eb83c9b8086"} Sep 11 20:15:32.020426 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:32.020388 2546 scope.go:117] "RemoveContainer" containerID="9fcc169600bf41619f3f0ba85397d170e17fb5e16ce156ba0c4c9e276425b6a6" Sep 11 20:15:32.032023 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:32.032003 2546 scope.go:117] "RemoveContainer" containerID="a56018aa4635859aa05bea293ab1a40edf5e86b901097789cf63f091dca4b35c" Sep 11 20:15:32.040949 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:32.040931 2546 scope.go:117] "RemoveContainer" containerID="9fcc169600bf41619f3f0ba85397d170e17fb5e16ce156ba0c4c9e276425b6a6" Sep 11 20:15:32.041213 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:15:32.041196 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fcc169600bf41619f3f0ba85397d170e17fb5e16ce156ba0c4c9e276425b6a6\": container with ID starting with 9fcc169600bf41619f3f0ba85397d170e17fb5e16ce156ba0c4c9e276425b6a6 not found: ID does not exist" containerID="9fcc169600bf41619f3f0ba85397d170e17fb5e16ce156ba0c4c9e276425b6a6" Sep 11 20:15:32.041276 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:32.041221 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fcc169600bf41619f3f0ba85397d170e17fb5e16ce156ba0c4c9e276425b6a6"} err="failed to get container status \"9fcc169600bf41619f3f0ba85397d170e17fb5e16ce156ba0c4c9e276425b6a6\": rpc error: code = NotFound desc = could not find container \"9fcc169600bf41619f3f0ba85397d170e17fb5e16ce156ba0c4c9e276425b6a6\": container with ID starting with 9fcc169600bf41619f3f0ba85397d170e17fb5e16ce156ba0c4c9e276425b6a6 not found: ID does not exist" Sep 11 20:15:32.041276 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:32.041236 2546 scope.go:117] "RemoveContainer" containerID="a56018aa4635859aa05bea293ab1a40edf5e86b901097789cf63f091dca4b35c" Sep 11 20:15:32.041485 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:15:32.041465 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a56018aa4635859aa05bea293ab1a40edf5e86b901097789cf63f091dca4b35c\": container with ID starting with a56018aa4635859aa05bea293ab1a40edf5e86b901097789cf63f091dca4b35c not found: ID does not exist" containerID="a56018aa4635859aa05bea293ab1a40edf5e86b901097789cf63f091dca4b35c" Sep 11 20:15:32.041546 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:32.041495 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a56018aa4635859aa05bea293ab1a40edf5e86b901097789cf63f091dca4b35c"} err="failed to get container status \"a56018aa4635859aa05bea293ab1a40edf5e86b901097789cf63f091dca4b35c\": rpc error: code = NotFound desc = could not find container \"a56018aa4635859aa05bea293ab1a40edf5e86b901097789cf63f091dca4b35c\": container with ID starting with a56018aa4635859aa05bea293ab1a40edf5e86b901097789cf63f091dca4b35c not found: ID does not exist" Sep 11 20:15:32.071198 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:32.071173 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eb3eda5-3f3d-4345-87f1-b24b1fd719bb-workload-certs\") pod \"3eb3eda5-3f3d-4345-87f1-b24b1fd719bb\" (UID: \"3eb3eda5-3f3d-4345-87f1-b24b1fd719bb\") " Sep 11 20:15:32.071303 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:32.071227 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb3eda5-3f3d-4345-87f1-b24b1fd719bb-credential-socket\") pod \"3eb3eda5-3f3d-4345-87f1-b24b1fd719bb\" (UID: \"3eb3eda5-3f3d-4345-87f1-b24b1fd719bb\") " Sep 11 20:15:32.071303 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:32.071260 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eb3eda5-3f3d-4345-87f1-b24b1fd719bb-istio-data\") pod \"3eb3eda5-3f3d-4345-87f1-b24b1fd719bb\" (UID: \"3eb3eda5-3f3d-4345-87f1-b24b1fd719bb\") " Sep 11 20:15:32.071303 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:32.071303 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eb3eda5-3f3d-4345-87f1-b24b1fd719bb-istio-podinfo\") pod \"3eb3eda5-3f3d-4345-87f1-b24b1fd719bb\" (UID: \"3eb3eda5-3f3d-4345-87f1-b24b1fd719bb\") " Sep 11 20:15:32.071480 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:32.071340 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k4cs7\" (UniqueName: \"kubernetes.io/projected/3eb3eda5-3f3d-4345-87f1-b24b1fd719bb-kube-api-access-k4cs7\") pod \"3eb3eda5-3f3d-4345-87f1-b24b1fd719bb\" (UID: \"3eb3eda5-3f3d-4345-87f1-b24b1fd719bb\") " Sep 11 20:15:32.071480 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:32.071404 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eb3eda5-3f3d-4345-87f1-b24b1fd719bb-istio-envoy\") pod \"3eb3eda5-3f3d-4345-87f1-b24b1fd719bb\" (UID: \"3eb3eda5-3f3d-4345-87f1-b24b1fd719bb\") " Sep 11 20:15:32.071604 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:32.071474 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb3eda5-3f3d-4345-87f1-b24b1fd719bb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3eb3eda5-3f3d-4345-87f1-b24b1fd719bb" (UID: "3eb3eda5-3f3d-4345-87f1-b24b1fd719bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:15:32.071671 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:32.071567 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb3eda5-3f3d-4345-87f1-b24b1fd719bb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3eb3eda5-3f3d-4345-87f1-b24b1fd719bb" (UID: "3eb3eda5-3f3d-4345-87f1-b24b1fd719bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:15:32.071671 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:32.071604 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb3eda5-3f3d-4345-87f1-b24b1fd719bb-istio-data" (OuterVolumeSpecName: "istio-data") pod "3eb3eda5-3f3d-4345-87f1-b24b1fd719bb" (UID: "3eb3eda5-3f3d-4345-87f1-b24b1fd719bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:15:32.071784 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:32.071677 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb3eda5-3f3d-4345-87f1-b24b1fd719bb-workload-socket\") pod \"3eb3eda5-3f3d-4345-87f1-b24b1fd719bb\" (UID: \"3eb3eda5-3f3d-4345-87f1-b24b1fd719bb\") " Sep 11 20:15:32.071784 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:32.071719 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eb3eda5-3f3d-4345-87f1-b24b1fd719bb-istio-token\") pod \"3eb3eda5-3f3d-4345-87f1-b24b1fd719bb\" (UID: \"3eb3eda5-3f3d-4345-87f1-b24b1fd719bb\") " Sep 11 20:15:32.071784 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:32.071755 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eb3eda5-3f3d-4345-87f1-b24b1fd719bb-istiod-ca-cert\") pod \"3eb3eda5-3f3d-4345-87f1-b24b1fd719bb\" (UID: \"3eb3eda5-3f3d-4345-87f1-b24b1fd719bb\") " Sep 11 20:15:32.071940 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:32.071904 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb3eda5-3f3d-4345-87f1-b24b1fd719bb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3eb3eda5-3f3d-4345-87f1-b24b1fd719bb" (UID: "3eb3eda5-3f3d-4345-87f1-b24b1fd719bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:15:32.072115 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:32.072084 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eb3eda5-3f3d-4345-87f1-b24b1fd719bb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3eb3eda5-3f3d-4345-87f1-b24b1fd719bb" (UID: "3eb3eda5-3f3d-4345-87f1-b24b1fd719bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:15:32.072222 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:32.072204 2546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb3eda5-3f3d-4345-87f1-b24b1fd719bb-workload-socket\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:15:32.072284 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:32.072228 2546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eb3eda5-3f3d-4345-87f1-b24b1fd719bb-istiod-ca-cert\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:15:32.072284 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:32.072243 2546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eb3eda5-3f3d-4345-87f1-b24b1fd719bb-workload-certs\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:15:32.072284 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:32.072256 2546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb3eda5-3f3d-4345-87f1-b24b1fd719bb-credential-socket\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:15:32.072284 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:32.072271 2546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eb3eda5-3f3d-4345-87f1-b24b1fd719bb-istio-data\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:15:32.073666 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:32.073636 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3eb3eda5-3f3d-4345-87f1-b24b1fd719bb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3eb3eda5-3f3d-4345-87f1-b24b1fd719bb" (UID: "3eb3eda5-3f3d-4345-87f1-b24b1fd719bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 20:15:32.073770 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:32.073690 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eb3eda5-3f3d-4345-87f1-b24b1fd719bb-kube-api-access-k4cs7" (OuterVolumeSpecName: "kube-api-access-k4cs7") pod "3eb3eda5-3f3d-4345-87f1-b24b1fd719bb" (UID: "3eb3eda5-3f3d-4345-87f1-b24b1fd719bb"). InnerVolumeSpecName "kube-api-access-k4cs7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:15:32.073819 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:32.073802 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eb3eda5-3f3d-4345-87f1-b24b1fd719bb-istio-token" (OuterVolumeSpecName: "istio-token") pod "3eb3eda5-3f3d-4345-87f1-b24b1fd719bb" (UID: "3eb3eda5-3f3d-4345-87f1-b24b1fd719bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:15:32.073863 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:32.073803 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb3eda5-3f3d-4345-87f1-b24b1fd719bb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3eb3eda5-3f3d-4345-87f1-b24b1fd719bb" (UID: "3eb3eda5-3f3d-4345-87f1-b24b1fd719bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:15:32.172899 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:32.172863 2546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eb3eda5-3f3d-4345-87f1-b24b1fd719bb-istio-envoy\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:15:32.172899 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:32.172893 2546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eb3eda5-3f3d-4345-87f1-b24b1fd719bb-istio-token\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:15:32.172899 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:32.172902 2546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eb3eda5-3f3d-4345-87f1-b24b1fd719bb-istio-podinfo\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:15:32.173129 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:32.172911 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4cs7\" (UniqueName: \"kubernetes.io/projected/3eb3eda5-3f3d-4345-87f1-b24b1fd719bb-kube-api-access-k4cs7\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:15:32.342110 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:32.342078 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795"] Sep 11 20:15:32.348738 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:32.348713 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-f5b4f720-openshift-default-6579fb5bd-8v795"] Sep 11 20:15:32.397589 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:32.397552 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:15:32.450711 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:32.450681 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3eb3eda5-3f3d-4345-87f1-b24b1fd719bb" path="/var/lib/kubelet/pods/3eb3eda5-3f3d-4345-87f1-b24b1fd719bb/volumes" Sep 11 20:15:32.451116 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:32.451103 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed23046d-f478-42bd-a143-f0add41d8342" path="/var/lib/kubelet/pods/ed23046d-f478-42bd-a143-f0add41d8342/volumes" Sep 11 20:15:33.242035 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:33.242005 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:15:34.018471 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:34.018438 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:15:34.800909 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:34.800875 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:15:35.022156 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:35.022119 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:15:35.124519 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:35.124491 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:15:35.224344 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:35.224301 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:15:35.325480 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:35.325445 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:15:35.653530 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:35.653497 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:15:35.809982 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:35.809946 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:15:35.907337 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:35.907233 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:15:43.027433 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:15:43.027403 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:16:01.570618 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:01.570571 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:16:05.114547 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:05.114511 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:16:07.425394 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:07.425343 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:16:07.529120 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:07.529087 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:16:07.626077 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:07.626049 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:16:12.761732 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:12.761697 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:16:12.993247 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:12.993216 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:16:13.409891 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:13.409857 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:16:13.699848 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:13.699749 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:16:13.809272 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:13.809170 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:16:16.109516 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:16.109482 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:16:16.902824 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:16.902788 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:16:16.981199 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:16.981162 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:16:17.778765 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:17.778733 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:16:18.735426 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:18.735390 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:16:19.421649 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:19.421621 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:16:20.221096 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:20.221065 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:16:20.307647 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:20.307617 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:16:20.703646 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:20.703607 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:16:20.802662 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:20.802635 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:16:20.820172 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:20.820143 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-3c2adb-5d9448cfc8-7qrs4"] Sep 11 20:16:20.820620 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:20.820608 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3eb3eda5-3f3d-4345-87f1-b24b1fd719bb" containerName="istio-proxy" Sep 11 20:16:20.820678 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:20.820622 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="3eb3eda5-3f3d-4345-87f1-b24b1fd719bb" containerName="istio-proxy" Sep 11 20:16:20.820678 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:20.820635 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed23046d-f478-42bd-a143-f0add41d8342" containerName="maas-api" Sep 11 20:16:20.820678 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:20.820641 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed23046d-f478-42bd-a143-f0add41d8342" containerName="maas-api" Sep 11 20:16:20.820678 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:20.820656 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6699d0e0-fd61-4739-82a2-30e1b96e2411" containerName="payload-pre-processing" Sep 11 20:16:20.820678 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:20.820662 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="6699d0e0-fd61-4739-82a2-30e1b96e2411" containerName="payload-pre-processing" Sep 11 20:16:20.820678 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:20.820670 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3eb3eda5-3f3d-4345-87f1-b24b1fd719bb" containerName="istio-proxy" Sep 11 20:16:20.820678 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:20.820676 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="3eb3eda5-3f3d-4345-87f1-b24b1fd719bb" containerName="istio-proxy" Sep 11 20:16:20.820892 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:20.820688 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51699108-a2b1-4bdb-bede-c7020defe411" containerName="payload-processing" Sep 11 20:16:20.820892 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:20.820693 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="51699108-a2b1-4bdb-bede-c7020defe411" containerName="payload-processing" Sep 11 20:16:20.820892 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:20.820755 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="3eb3eda5-3f3d-4345-87f1-b24b1fd719bb" containerName="istio-proxy" Sep 11 20:16:20.820892 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:20.820763 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="51699108-a2b1-4bdb-bede-c7020defe411" containerName="payload-processing" Sep 11 20:16:20.820892 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:20.820771 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed23046d-f478-42bd-a143-f0add41d8342" containerName="maas-api" Sep 11 20:16:20.820892 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:20.820778 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="6699d0e0-fd61-4739-82a2-30e1b96e2411" containerName="payload-pre-processing" Sep 11 20:16:20.823961 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:20.823942 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3c2adb-5d9448cfc8-7qrs4" Sep 11 20:16:20.826407 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:20.826388 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-3c2adb\"" Sep 11 20:16:20.826535 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:20.826409 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-3c2adb-dockercfg-8mm2t\"" Sep 11 20:16:20.836369 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:20.836347 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-3c2adb-6685c45545-vft65"] Sep 11 20:16:20.837040 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:20.837021 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="3eb3eda5-3f3d-4345-87f1-b24b1fd719bb" containerName="istio-proxy" Sep 11 20:16:20.840436 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:20.840420 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-3c2adb-5d9448cfc8-7qrs4"] Sep 11 20:16:20.840533 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:20.840500 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-3c2adb-6685c45545-vft65" Sep 11 20:16:20.854855 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:20.854831 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-3c2adb-6685c45545-vft65"] Sep 11 20:16:20.945643 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:20.945610 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgn5m\" (UniqueName: \"kubernetes.io/projected/93fbc9a3-6d26-4302-b31e-a0427d0895d7-kube-api-access-cgn5m\") pod \"payload-processing-e2e-explicit-3c2adb-6685c45545-vft65\" (UID: \"93fbc9a3-6d26-4302-b31e-a0427d0895d7\") " pod="openshift-ingress/payload-processing-e2e-explicit-3c2adb-6685c45545-vft65" Sep 11 20:16:20.945829 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:20.945653 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d9kt\" (UniqueName: \"kubernetes.io/projected/c889f806-42e1-418c-8b3b-054e4f0b1250-kube-api-access-9d9kt\") pod \"payload-pre-processing-e2e-explicit-3c2adb-5d9448cfc8-7qrs4\" (UID: \"c889f806-42e1-418c-8b3b-054e4f0b1250\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-3c2adb-5d9448cfc8-7qrs4" Sep 11 20:16:20.945829 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:20.945789 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c889f806-42e1-418c-8b3b-054e4f0b1250-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-3c2adb-5d9448cfc8-7qrs4\" (UID: \"c889f806-42e1-418c-8b3b-054e4f0b1250\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-3c2adb-5d9448cfc8-7qrs4" Sep 11 20:16:20.945950 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:20.945833 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93fbc9a3-6d26-4302-b31e-a0427d0895d7-plugins-config-volume\") pod \"payload-processing-e2e-explicit-3c2adb-6685c45545-vft65\" (UID: \"93fbc9a3-6d26-4302-b31e-a0427d0895d7\") " pod="openshift-ingress/payload-processing-e2e-explicit-3c2adb-6685c45545-vft65" Sep 11 20:16:21.047067 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:21.046982 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgn5m\" (UniqueName: \"kubernetes.io/projected/93fbc9a3-6d26-4302-b31e-a0427d0895d7-kube-api-access-cgn5m\") pod \"payload-processing-e2e-explicit-3c2adb-6685c45545-vft65\" (UID: \"93fbc9a3-6d26-4302-b31e-a0427d0895d7\") " pod="openshift-ingress/payload-processing-e2e-explicit-3c2adb-6685c45545-vft65" Sep 11 20:16:21.047067 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:21.047026 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d9kt\" (UniqueName: \"kubernetes.io/projected/c889f806-42e1-418c-8b3b-054e4f0b1250-kube-api-access-9d9kt\") pod \"payload-pre-processing-e2e-explicit-3c2adb-5d9448cfc8-7qrs4\" (UID: \"c889f806-42e1-418c-8b3b-054e4f0b1250\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-3c2adb-5d9448cfc8-7qrs4" Sep 11 20:16:21.047298 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:21.047088 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c889f806-42e1-418c-8b3b-054e4f0b1250-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-3c2adb-5d9448cfc8-7qrs4\" (UID: \"c889f806-42e1-418c-8b3b-054e4f0b1250\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-3c2adb-5d9448cfc8-7qrs4" Sep 11 20:16:21.047298 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:21.047108 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93fbc9a3-6d26-4302-b31e-a0427d0895d7-plugins-config-volume\") pod \"payload-processing-e2e-explicit-3c2adb-6685c45545-vft65\" (UID: \"93fbc9a3-6d26-4302-b31e-a0427d0895d7\") " pod="openshift-ingress/payload-processing-e2e-explicit-3c2adb-6685c45545-vft65" Sep 11 20:16:21.047870 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:21.047848 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c889f806-42e1-418c-8b3b-054e4f0b1250-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-3c2adb-5d9448cfc8-7qrs4\" (UID: \"c889f806-42e1-418c-8b3b-054e4f0b1250\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-3c2adb-5d9448cfc8-7qrs4" Sep 11 20:16:21.047937 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:21.047870 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93fbc9a3-6d26-4302-b31e-a0427d0895d7-plugins-config-volume\") pod \"payload-processing-e2e-explicit-3c2adb-6685c45545-vft65\" (UID: \"93fbc9a3-6d26-4302-b31e-a0427d0895d7\") " pod="openshift-ingress/payload-processing-e2e-explicit-3c2adb-6685c45545-vft65" Sep 11 20:16:21.055151 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:21.055130 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d9kt\" (UniqueName: \"kubernetes.io/projected/c889f806-42e1-418c-8b3b-054e4f0b1250-kube-api-access-9d9kt\") pod \"payload-pre-processing-e2e-explicit-3c2adb-5d9448cfc8-7qrs4\" (UID: \"c889f806-42e1-418c-8b3b-054e4f0b1250\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-3c2adb-5d9448cfc8-7qrs4" Sep 11 20:16:21.055246 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:21.055169 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgn5m\" (UniqueName: \"kubernetes.io/projected/93fbc9a3-6d26-4302-b31e-a0427d0895d7-kube-api-access-cgn5m\") pod \"payload-processing-e2e-explicit-3c2adb-6685c45545-vft65\" (UID: \"93fbc9a3-6d26-4302-b31e-a0427d0895d7\") " pod="openshift-ingress/payload-processing-e2e-explicit-3c2adb-6685c45545-vft65" Sep 11 20:16:21.128186 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:21.128162 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:16:21.138274 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:21.138246 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3c2adb-5d9448cfc8-7qrs4" Sep 11 20:16:21.153558 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:21.153536 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-3c2adb-6685c45545-vft65" Sep 11 20:16:21.417911 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:21.417882 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-3c2adb-5d9448cfc8-7qrs4"] Sep 11 20:16:21.418980 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:16:21.418956 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc889f806_42e1_418c_8b3b_054e4f0b1250.slice/crio-6c330cf0489fb75a08bb322321ec2b07a8ec328d43b9fdbead1f2aa82b0cab27 WatchSource:0}: Error finding container 6c330cf0489fb75a08bb322321ec2b07a8ec328d43b9fdbead1f2aa82b0cab27: Status 404 returned error can't find the container with id 6c330cf0489fb75a08bb322321ec2b07a8ec328d43b9fdbead1f2aa82b0cab27 Sep 11 20:16:21.438630 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:21.438606 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-3c2adb-6685c45545-vft65"] Sep 11 20:16:21.440208 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:16:21.440179 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93fbc9a3_6d26_4302_b31e_a0427d0895d7.slice/crio-fb3fbff3a852c528dd23b1dfbb3d9b9c7e89d2296d726cfcb3a027cd6431c5d4 WatchSource:0}: Error finding container fb3fbff3a852c528dd23b1dfbb3d9b9c7e89d2296d726cfcb3a027cd6431c5d4: Status 404 returned error can't find the container with id fb3fbff3a852c528dd23b1dfbb3d9b9c7e89d2296d726cfcb3a027cd6431c5d4 Sep 11 20:16:22.296036 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:22.296003 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-3c2adb-6685c45545-vft65" event={"ID":"93fbc9a3-6d26-4302-b31e-a0427d0895d7","Type":"ContainerStarted","Data":"ef74047c1d99f6e10cb24a0979cb58a6114e547992d96e4057d608bed31b32ee"} Sep 11 20:16:22.296494 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:22.296054 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-3c2adb-6685c45545-vft65" event={"ID":"93fbc9a3-6d26-4302-b31e-a0427d0895d7","Type":"ContainerStarted","Data":"fb3fbff3a852c528dd23b1dfbb3d9b9c7e89d2296d726cfcb3a027cd6431c5d4"} Sep 11 20:16:22.296494 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:22.296076 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-3c2adb-6685c45545-vft65" Sep 11 20:16:22.297678 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:22.297652 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3c2adb-5d9448cfc8-7qrs4" event={"ID":"c889f806-42e1-418c-8b3b-054e4f0b1250","Type":"ContainerStarted","Data":"11a771ad2d58248960a70e02ceb784322917697afe54a62c0b045b7eb7dfcd5e"} Sep 11 20:16:22.297885 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:22.297702 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3c2adb-5d9448cfc8-7qrs4" event={"ID":"c889f806-42e1-418c-8b3b-054e4f0b1250","Type":"ContainerStarted","Data":"6c330cf0489fb75a08bb322321ec2b07a8ec328d43b9fdbead1f2aa82b0cab27"} Sep 11 20:16:22.297885 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:22.297735 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3c2adb-5d9448cfc8-7qrs4" Sep 11 20:16:22.314382 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:22.314336 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-3c2adb-6685c45545-vft65" podStartSLOduration=2.314321292 podStartE2EDuration="2.314321292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:16:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:16:22.311883667 +0000 UTC m=+952.359934012" watchObservedRunningTime="2026-09-11 20:16:22.314321292 +0000 UTC m=+952.362371638" Sep 11 20:16:22.329830 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:22.329779 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3c2adb-5d9448cfc8-7qrs4" podStartSLOduration=2.329762652 podStartE2EDuration="2.329762652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:16:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:16:22.326778248 +0000 UTC m=+952.374828598" watchObservedRunningTime="2026-09-11 20:16:22.329762652 +0000 UTC m=+952.377813001" Sep 11 20:16:22.376179 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:22.376153 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:16:22.434687 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:22.434646 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:16:23.416404 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:23.416373 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:16:29.201951 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:29.201914 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:16:29.316527 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:29.316499 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:16:29.401898 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:29.401858 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:16:33.305529 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:33.305499 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-3c2adb-6685c45545-vft65" Sep 11 20:16:33.305914 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:33.305567 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3c2adb-5d9448cfc8-7qrs4" Sep 11 20:16:39.831463 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:39.831422 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:16:40.569422 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:40.569391 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:16:43.138489 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:43.138453 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:16:43.874918 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:43.874883 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:16:44.576913 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:44.576883 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:16:45.361738 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:45.361701 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:16:46.198735 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:46.198699 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:16:46.737672 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:46.737642 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:16:46.846156 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:46.846106 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:16:46.938474 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:46.938434 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:16:47.034297 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:47.034223 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:16:47.115198 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:47.115163 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:16:47.600968 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:47.600928 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:16:47.776933 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:47.776901 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:16:47.879372 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:47.879300 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:16:49.204426 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:49.204391 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:16:50.608988 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:50.608948 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:16:54.919768 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:54.919683 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:16:54.981619 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:54.981591 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:16:55.565034 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:16:55.565001 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:17:01.604349 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:01.604311 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:17:01.751229 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:01.751196 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:17:01.904261 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:01.904183 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:17:02.186267 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:02.186185 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:17:02.504211 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:02.504129 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:17:07.412418 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:07.412387 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:17:08.141385 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:08.141349 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:17:08.239804 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:08.239771 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:17:09.226694 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:09.226661 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:17:10.127429 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:10.127394 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:17:11.011685 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:11.011649 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:17:11.662717 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:11.662686 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:17:12.342378 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:12.342334 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:17:12.406260 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:12.406226 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:17:12.540258 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:12.540228 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:17:12.902191 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:12.902162 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:17:13.007111 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:13.007077 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:17:13.008623 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:13.008600 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6344a120-7f8cddc646-56rql"] Sep 11 20:17:13.012684 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:13.012669 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6344a120-7f8cddc646-56rql" Sep 11 20:17:13.015071 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:13.015048 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-6344a120-dockercfg-kv4fw\"" Sep 11 20:17:13.015183 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:13.015118 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-6344a120\"" Sep 11 20:17:13.021205 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:13.020903 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6344a120-7f8cddc646-56rql"] Sep 11 20:17:13.042362 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:13.042336 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6344a120-b94b4c697-x9f6x"] Sep 11 20:17:13.046121 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:13.046104 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6344a120-b94b4c697-x9f6x" Sep 11 20:17:13.055231 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:13.055206 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6344a120-b94b4c697-x9f6x"] Sep 11 20:17:13.074983 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:13.074958 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d86ac50-085a-47a7-94aa-95ecbee3bd3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6344a120-7f8cddc646-56rql\" (UID: \"0d86ac50-085a-47a7-94aa-95ecbee3bd3c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6344a120-7f8cddc646-56rql" Sep 11 20:17:13.075088 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:13.074992 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k89xv\" (UniqueName: \"kubernetes.io/projected/642e7b32-0972-49c8-8b03-9798bb7d0838-kube-api-access-k89xv\") pod \"payload-processing-e2e-mt-6344a120-b94b4c697-x9f6x\" (UID: \"642e7b32-0972-49c8-8b03-9798bb7d0838\") " pod="openshift-ingress/payload-processing-e2e-mt-6344a120-b94b4c697-x9f6x" Sep 11 20:17:13.075088 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:13.075025 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/642e7b32-0972-49c8-8b03-9798bb7d0838-plugins-config-volume\") pod \"payload-processing-e2e-mt-6344a120-b94b4c697-x9f6x\" (UID: \"642e7b32-0972-49c8-8b03-9798bb7d0838\") " pod="openshift-ingress/payload-processing-e2e-mt-6344a120-b94b4c697-x9f6x" Sep 11 20:17:13.075088 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:13.075043 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfswr\" (UniqueName: \"kubernetes.io/projected/0d86ac50-085a-47a7-94aa-95ecbee3bd3c-kube-api-access-lfswr\") pod \"payload-pre-processing-e2e-mt-6344a120-7f8cddc646-56rql\" (UID: \"0d86ac50-085a-47a7-94aa-95ecbee3bd3c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6344a120-7f8cddc646-56rql" Sep 11 20:17:13.175768 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:13.175688 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d86ac50-085a-47a7-94aa-95ecbee3bd3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6344a120-7f8cddc646-56rql\" (UID: \"0d86ac50-085a-47a7-94aa-95ecbee3bd3c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6344a120-7f8cddc646-56rql" Sep 11 20:17:13.175768 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:13.175734 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k89xv\" (UniqueName: \"kubernetes.io/projected/642e7b32-0972-49c8-8b03-9798bb7d0838-kube-api-access-k89xv\") pod \"payload-processing-e2e-mt-6344a120-b94b4c697-x9f6x\" (UID: \"642e7b32-0972-49c8-8b03-9798bb7d0838\") " pod="openshift-ingress/payload-processing-e2e-mt-6344a120-b94b4c697-x9f6x" Sep 11 20:17:13.176034 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:13.175791 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/642e7b32-0972-49c8-8b03-9798bb7d0838-plugins-config-volume\") pod \"payload-processing-e2e-mt-6344a120-b94b4c697-x9f6x\" (UID: \"642e7b32-0972-49c8-8b03-9798bb7d0838\") " pod="openshift-ingress/payload-processing-e2e-mt-6344a120-b94b4c697-x9f6x" Sep 11 20:17:13.176034 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:13.175819 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfswr\" (UniqueName: \"kubernetes.io/projected/0d86ac50-085a-47a7-94aa-95ecbee3bd3c-kube-api-access-lfswr\") pod \"payload-pre-processing-e2e-mt-6344a120-7f8cddc646-56rql\" (UID: \"0d86ac50-085a-47a7-94aa-95ecbee3bd3c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6344a120-7f8cddc646-56rql" Sep 11 20:17:13.176477 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:13.176453 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d86ac50-085a-47a7-94aa-95ecbee3bd3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6344a120-7f8cddc646-56rql\" (UID: \"0d86ac50-085a-47a7-94aa-95ecbee3bd3c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6344a120-7f8cddc646-56rql" Sep 11 20:17:13.176605 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:13.176512 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/642e7b32-0972-49c8-8b03-9798bb7d0838-plugins-config-volume\") pod \"payload-processing-e2e-mt-6344a120-b94b4c697-x9f6x\" (UID: \"642e7b32-0972-49c8-8b03-9798bb7d0838\") " pod="openshift-ingress/payload-processing-e2e-mt-6344a120-b94b4c697-x9f6x" Sep 11 20:17:13.183697 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:13.183676 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k89xv\" (UniqueName: \"kubernetes.io/projected/642e7b32-0972-49c8-8b03-9798bb7d0838-kube-api-access-k89xv\") pod \"payload-processing-e2e-mt-6344a120-b94b4c697-x9f6x\" (UID: \"642e7b32-0972-49c8-8b03-9798bb7d0838\") " pod="openshift-ingress/payload-processing-e2e-mt-6344a120-b94b4c697-x9f6x" Sep 11 20:17:13.183834 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:13.183812 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfswr\" (UniqueName: \"kubernetes.io/projected/0d86ac50-085a-47a7-94aa-95ecbee3bd3c-kube-api-access-lfswr\") pod \"payload-pre-processing-e2e-mt-6344a120-7f8cddc646-56rql\" (UID: \"0d86ac50-085a-47a7-94aa-95ecbee3bd3c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6344a120-7f8cddc646-56rql" Sep 11 20:17:13.325847 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:13.325791 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6344a120-7f8cddc646-56rql" Sep 11 20:17:13.357519 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:13.357490 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6344a120-b94b4c697-x9f6x" Sep 11 20:17:13.605119 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:13.605093 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6344a120-7f8cddc646-56rql"] Sep 11 20:17:13.606704 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:17:13.606676 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d86ac50_085a_47a7_94aa_95ecbee3bd3c.slice/crio-c664f21746be93026f04539c5b15bd3fec90f0405e5c923b29b1d13116730da5 WatchSource:0}: Error finding container c664f21746be93026f04539c5b15bd3fec90f0405e5c923b29b1d13116730da5: Status 404 returned error can't find the container with id c664f21746be93026f04539c5b15bd3fec90f0405e5c923b29b1d13116730da5 Sep 11 20:17:13.639279 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:13.639256 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6344a120-b94b4c697-x9f6x"] Sep 11 20:17:13.643704 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:17:13.643677 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod642e7b32_0972_49c8_8b03_9798bb7d0838.slice/crio-78fa77c26dcab0ce1d9230b32762c5d17912588220806737d10354761b1410ee WatchSource:0}: Error finding container 78fa77c26dcab0ce1d9230b32762c5d17912588220806737d10354761b1410ee: Status 404 returned error can't find the container with id 78fa77c26dcab0ce1d9230b32762c5d17912588220806737d10354761b1410ee Sep 11 20:17:14.601004 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:14.600970 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6344a120-b94b4c697-x9f6x" event={"ID":"642e7b32-0972-49c8-8b03-9798bb7d0838","Type":"ContainerStarted","Data":"2028c5e3d060dd3f9f21ed9c5cc36d485c4cf05f961db959e93fac96a69d53c6"} Sep 11 20:17:14.601488 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:14.601024 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6344a120-b94b4c697-x9f6x" event={"ID":"642e7b32-0972-49c8-8b03-9798bb7d0838","Type":"ContainerStarted","Data":"78fa77c26dcab0ce1d9230b32762c5d17912588220806737d10354761b1410ee"} Sep 11 20:17:14.601488 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:14.601091 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-6344a120-b94b4c697-x9f6x" Sep 11 20:17:14.602614 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:14.602593 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6344a120-7f8cddc646-56rql" event={"ID":"0d86ac50-085a-47a7-94aa-95ecbee3bd3c","Type":"ContainerStarted","Data":"b6171126bf93c6a239e48cdfc9ea398e812036920a5fe9f1e9b1b60f8d22e8cb"} Sep 11 20:17:14.602721 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:14.602633 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6344a120-7f8cddc646-56rql" event={"ID":"0d86ac50-085a-47a7-94aa-95ecbee3bd3c","Type":"ContainerStarted","Data":"c664f21746be93026f04539c5b15bd3fec90f0405e5c923b29b1d13116730da5"} Sep 11 20:17:14.602721 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:14.602669 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-6344a120-7f8cddc646-56rql" Sep 11 20:17:14.619430 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:14.619382 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-6344a120-b94b4c697-x9f6x" podStartSLOduration=1.619368286 podStartE2EDuration="1.619368286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:17:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:17:14.618253948 +0000 UTC m=+1004.666304292" watchObservedRunningTime="2026-09-11 20:17:14.619368286 +0000 UTC m=+1004.667418630" Sep 11 20:17:14.634948 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:14.634904 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-6344a120-7f8cddc646-56rql" podStartSLOduration=2.634891382 podStartE2EDuration="2.634891382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:17:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:17:14.633547805 +0000 UTC m=+1004.681598150" watchObservedRunningTime="2026-09-11 20:17:14.634891382 +0000 UTC m=+1004.682941726" Sep 11 20:17:25.609940 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:25.609905 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-6344a120-7f8cddc646-56rql" Sep 11 20:17:25.610403 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:25.610067 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-6344a120-b94b4c697-x9f6x" Sep 11 20:17:34.328406 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:34.328373 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:17:35.211733 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:35.211696 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:17:35.929340 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:35.929303 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:17:36.046092 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:36.046065 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:17:36.816675 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:36.816628 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:17:37.506837 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:37.506789 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:17:38.164356 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:38.164325 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:17:38.316723 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:38.316694 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:17:38.817616 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:38.817557 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:17:38.914862 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:38.914828 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:17:39.113018 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:39.112984 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:17:39.210757 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:39.210718 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:17:44.261436 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:44.261403 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:17:45.148284 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:45.148245 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:17:45.239356 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:45.239325 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:17:46.429760 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:46.429706 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:17:47.119311 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:47.119274 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:17:47.917645 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:47.917615 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:17:48.603409 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:48.603378 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:17:49.404644 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:49.404609 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:17:50.104809 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:50.104777 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:17:51.002002 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:51.001972 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:17:56.028765 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:56.028731 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:17:56.052603 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:56.052539 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6344a120-b94b4c697-x9f6x"] Sep 11 20:17:56.052970 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:56.052795 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-6344a120-b94b4c697-x9f6x" podUID="642e7b32-0972-49c8-8b03-9798bb7d0838" containerName="payload-processing" containerID="cri-o://2028c5e3d060dd3f9f21ed9c5cc36d485c4cf05f961db959e93fac96a69d53c6" gracePeriod=30 Sep 11 20:17:56.060809 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:56.060317 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6344a120-7f8cddc646-56rql"] Sep 11 20:17:56.060809 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:56.060522 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-6344a120-7f8cddc646-56rql" podUID="0d86ac50-085a-47a7-94aa-95ecbee3bd3c" containerName="payload-pre-processing" containerID="cri-o://b6171126bf93c6a239e48cdfc9ea398e812036920a5fe9f1e9b1b60f8d22e8cb" gracePeriod=30 Sep 11 20:17:56.094938 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:56.094908 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:17:56.184405 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:56.184379 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:17:56.273065 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:56.272983 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:17:56.339198 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:56.339164 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6344a120-b94b4c697-x9f6x" Sep 11 20:17:56.391108 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:56.391083 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6344a120-7f8cddc646-56rql" Sep 11 20:17:56.422681 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:56.422653 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/642e7b32-0972-49c8-8b03-9798bb7d0838-plugins-config-volume\") pod \"642e7b32-0972-49c8-8b03-9798bb7d0838\" (UID: \"642e7b32-0972-49c8-8b03-9798bb7d0838\") " Sep 11 20:17:56.422830 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:56.422743 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k89xv\" (UniqueName: \"kubernetes.io/projected/642e7b32-0972-49c8-8b03-9798bb7d0838-kube-api-access-k89xv\") pod \"642e7b32-0972-49c8-8b03-9798bb7d0838\" (UID: \"642e7b32-0972-49c8-8b03-9798bb7d0838\") " Sep 11 20:17:56.423034 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:56.423012 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/642e7b32-0972-49c8-8b03-9798bb7d0838-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "642e7b32-0972-49c8-8b03-9798bb7d0838" (UID: "642e7b32-0972-49c8-8b03-9798bb7d0838"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:17:56.424955 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:56.424919 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/642e7b32-0972-49c8-8b03-9798bb7d0838-kube-api-access-k89xv" (OuterVolumeSpecName: "kube-api-access-k89xv") pod "642e7b32-0972-49c8-8b03-9798bb7d0838" (UID: "642e7b32-0972-49c8-8b03-9798bb7d0838"). InnerVolumeSpecName "kube-api-access-k89xv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:17:56.524101 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:56.524073 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d86ac50-085a-47a7-94aa-95ecbee3bd3c-plugins-config-volume\") pod \"0d86ac50-085a-47a7-94aa-95ecbee3bd3c\" (UID: \"0d86ac50-085a-47a7-94aa-95ecbee3bd3c\") " Sep 11 20:17:56.524225 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:56.524204 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lfswr\" (UniqueName: \"kubernetes.io/projected/0d86ac50-085a-47a7-94aa-95ecbee3bd3c-kube-api-access-lfswr\") pod \"0d86ac50-085a-47a7-94aa-95ecbee3bd3c\" (UID: \"0d86ac50-085a-47a7-94aa-95ecbee3bd3c\") " Sep 11 20:17:56.524427 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:56.524400 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d86ac50-085a-47a7-94aa-95ecbee3bd3c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0d86ac50-085a-47a7-94aa-95ecbee3bd3c" (UID: "0d86ac50-085a-47a7-94aa-95ecbee3bd3c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:17:56.524655 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:56.524592 2546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d86ac50-085a-47a7-94aa-95ecbee3bd3c-plugins-config-volume\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:17:56.524655 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:56.524618 2546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/642e7b32-0972-49c8-8b03-9798bb7d0838-plugins-config-volume\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:17:56.524655 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:56.524633 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k89xv\" (UniqueName: \"kubernetes.io/projected/642e7b32-0972-49c8-8b03-9798bb7d0838-kube-api-access-k89xv\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:17:56.527034 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:56.527007 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d86ac50-085a-47a7-94aa-95ecbee3bd3c-kube-api-access-lfswr" (OuterVolumeSpecName: "kube-api-access-lfswr") pod "0d86ac50-085a-47a7-94aa-95ecbee3bd3c" (UID: "0d86ac50-085a-47a7-94aa-95ecbee3bd3c"). InnerVolumeSpecName "kube-api-access-lfswr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:17:56.541257 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:56.541198 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:17:56.626117 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:56.626086 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfswr\" (UniqueName: \"kubernetes.io/projected/0d86ac50-085a-47a7-94aa-95ecbee3bd3c-kube-api-access-lfswr\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:17:56.688025 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:56.687996 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:17:56.837156 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:56.837078 2546 generic.go:356] "Generic (PLEG): container finished" podID="642e7b32-0972-49c8-8b03-9798bb7d0838" containerID="2028c5e3d060dd3f9f21ed9c5cc36d485c4cf05f961db959e93fac96a69d53c6" exitCode=0 Sep 11 20:17:56.837156 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:56.837126 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6344a120-b94b4c697-x9f6x" Sep 11 20:17:56.837358 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:56.837168 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6344a120-b94b4c697-x9f6x" event={"ID":"642e7b32-0972-49c8-8b03-9798bb7d0838","Type":"ContainerDied","Data":"2028c5e3d060dd3f9f21ed9c5cc36d485c4cf05f961db959e93fac96a69d53c6"} Sep 11 20:17:56.837358 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:56.837230 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6344a120-b94b4c697-x9f6x" event={"ID":"642e7b32-0972-49c8-8b03-9798bb7d0838","Type":"ContainerDied","Data":"78fa77c26dcab0ce1d9230b32762c5d17912588220806737d10354761b1410ee"} Sep 11 20:17:56.837358 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:56.837260 2546 scope.go:117] "RemoveContainer" containerID="2028c5e3d060dd3f9f21ed9c5cc36d485c4cf05f961db959e93fac96a69d53c6" Sep 11 20:17:56.838618 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:56.838570 2546 generic.go:356] "Generic (PLEG): container finished" podID="0d86ac50-085a-47a7-94aa-95ecbee3bd3c" containerID="b6171126bf93c6a239e48cdfc9ea398e812036920a5fe9f1e9b1b60f8d22e8cb" exitCode=0 Sep 11 20:17:56.838715 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:56.838631 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6344a120-7f8cddc646-56rql" Sep 11 20:17:56.838715 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:56.838681 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6344a120-7f8cddc646-56rql" event={"ID":"0d86ac50-085a-47a7-94aa-95ecbee3bd3c","Type":"ContainerDied","Data":"b6171126bf93c6a239e48cdfc9ea398e812036920a5fe9f1e9b1b60f8d22e8cb"} Sep 11 20:17:56.838839 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:56.838732 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6344a120-7f8cddc646-56rql" event={"ID":"0d86ac50-085a-47a7-94aa-95ecbee3bd3c","Type":"ContainerDied","Data":"c664f21746be93026f04539c5b15bd3fec90f0405e5c923b29b1d13116730da5"} Sep 11 20:17:56.855706 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:56.855681 2546 scope.go:117] "RemoveContainer" containerID="2028c5e3d060dd3f9f21ed9c5cc36d485c4cf05f961db959e93fac96a69d53c6" Sep 11 20:17:56.856141 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:17:56.856117 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2028c5e3d060dd3f9f21ed9c5cc36d485c4cf05f961db959e93fac96a69d53c6\": container with ID starting with 2028c5e3d060dd3f9f21ed9c5cc36d485c4cf05f961db959e93fac96a69d53c6 not found: ID does not exist" containerID="2028c5e3d060dd3f9f21ed9c5cc36d485c4cf05f961db959e93fac96a69d53c6" Sep 11 20:17:56.856271 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:56.856148 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2028c5e3d060dd3f9f21ed9c5cc36d485c4cf05f961db959e93fac96a69d53c6"} err="failed to get container status \"2028c5e3d060dd3f9f21ed9c5cc36d485c4cf05f961db959e93fac96a69d53c6\": rpc error: code = NotFound desc = could not find container \"2028c5e3d060dd3f9f21ed9c5cc36d485c4cf05f961db959e93fac96a69d53c6\": container with ID starting with 2028c5e3d060dd3f9f21ed9c5cc36d485c4cf05f961db959e93fac96a69d53c6 not found: ID does not exist" Sep 11 20:17:56.856271 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:56.856165 2546 scope.go:117] "RemoveContainer" containerID="b6171126bf93c6a239e48cdfc9ea398e812036920a5fe9f1e9b1b60f8d22e8cb" Sep 11 20:17:56.857322 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:56.857240 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6344a120-b94b4c697-x9f6x"] Sep 11 20:17:56.859372 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:56.859347 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6344a120-b94b4c697-x9f6x"] Sep 11 20:17:56.864803 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:56.864786 2546 scope.go:117] "RemoveContainer" containerID="b6171126bf93c6a239e48cdfc9ea398e812036920a5fe9f1e9b1b60f8d22e8cb" Sep 11 20:17:56.865048 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:17:56.865031 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6171126bf93c6a239e48cdfc9ea398e812036920a5fe9f1e9b1b60f8d22e8cb\": container with ID starting with b6171126bf93c6a239e48cdfc9ea398e812036920a5fe9f1e9b1b60f8d22e8cb not found: ID does not exist" containerID="b6171126bf93c6a239e48cdfc9ea398e812036920a5fe9f1e9b1b60f8d22e8cb" Sep 11 20:17:56.865095 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:56.865054 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6171126bf93c6a239e48cdfc9ea398e812036920a5fe9f1e9b1b60f8d22e8cb"} err="failed to get container status \"b6171126bf93c6a239e48cdfc9ea398e812036920a5fe9f1e9b1b60f8d22e8cb\": rpc error: code = NotFound desc = could not find container \"b6171126bf93c6a239e48cdfc9ea398e812036920a5fe9f1e9b1b60f8d22e8cb\": container with ID starting with b6171126bf93c6a239e48cdfc9ea398e812036920a5fe9f1e9b1b60f8d22e8cb not found: ID does not exist" Sep 11 20:17:56.876923 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:56.876904 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6344a120-7f8cddc646-56rql"] Sep 11 20:17:56.880867 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:56.880849 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6344a120-7f8cddc646-56rql"] Sep 11 20:17:57.539456 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:57.539412 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:17:58.258768 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:58.258741 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:17:58.450503 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:58.450473 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d86ac50-085a-47a7-94aa-95ecbee3bd3c" path="/var/lib/kubelet/pods/0d86ac50-085a-47a7-94aa-95ecbee3bd3c/volumes" Sep 11 20:17:58.450896 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:58.450881 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="642e7b32-0972-49c8-8b03-9798bb7d0838" path="/var/lib/kubelet/pods/642e7b32-0972-49c8-8b03-9798bb7d0838/volumes" Sep 11 20:17:59.109836 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:59.109803 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:17:59.568569 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:59.568475 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:17:59.972055 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:17:59.972018 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:00.802403 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:00.802376 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:02.316706 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:02.316665 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:02.629244 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:02.629205 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:03.266705 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:03.266677 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:03.372200 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:03.372166 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:03.494896 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:03.494863 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:03.614046 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:03.614018 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:03.810555 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:03.810522 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:03.907049 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:03.906974 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:04.007966 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:04.007935 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:04.431834 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:04.431798 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:05.400715 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:05.400684 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:06.139892 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:06.139859 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:06.809613 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:06.809559 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:07.542307 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:07.542280 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:10.017208 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:10.017174 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:10.122363 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:10.122327 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:10.217147 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:10.217113 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:10.334869 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:10.334785 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:13.056046 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:13.056010 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:13.094002 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:13.093966 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-3c2adb-6685c45545-vft65"] Sep 11 20:18:13.094492 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:13.094461 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-3c2adb-6685c45545-vft65" podUID="93fbc9a3-6d26-4302-b31e-a0427d0895d7" containerName="payload-processing" containerID="cri-o://ef74047c1d99f6e10cb24a0979cb58a6114e547992d96e4057d608bed31b32ee" gracePeriod=30 Sep 11 20:18:13.103021 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:13.102831 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-3c2adb-5d9448cfc8-7qrs4"] Sep 11 20:18:13.103103 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:13.103039 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3c2adb-5d9448cfc8-7qrs4" podUID="c889f806-42e1-418c-8b3b-054e4f0b1250" containerName="payload-pre-processing" containerID="cri-o://11a771ad2d58248960a70e02ceb784322917697afe54a62c0b045b7eb7dfcd5e" gracePeriod=30 Sep 11 20:18:13.131607 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:13.131548 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:13.228798 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:13.228750 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:13.290131 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:13.290097 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:13.304420 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:13.304386 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3c2adb-5d9448cfc8-7qrs4" podUID="c889f806-42e1-418c-8b3b-054e4f0b1250" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.60:9004: connect: connection refused" Sep 11 20:18:13.395606 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:13.395566 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-3c2adb-6685c45545-vft65" Sep 11 20:18:13.398752 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:13.398727 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3c2adb-5d9448cfc8-7qrs4" Sep 11 20:18:13.491463 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:13.491436 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9d9kt\" (UniqueName: \"kubernetes.io/projected/c889f806-42e1-418c-8b3b-054e4f0b1250-kube-api-access-9d9kt\") pod \"c889f806-42e1-418c-8b3b-054e4f0b1250\" (UID: \"c889f806-42e1-418c-8b3b-054e4f0b1250\") " Sep 11 20:18:13.491691 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:13.491487 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c889f806-42e1-418c-8b3b-054e4f0b1250-plugins-config-volume\") pod \"c889f806-42e1-418c-8b3b-054e4f0b1250\" (UID: \"c889f806-42e1-418c-8b3b-054e4f0b1250\") " Sep 11 20:18:13.491691 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:13.491532 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93fbc9a3-6d26-4302-b31e-a0427d0895d7-plugins-config-volume\") pod \"93fbc9a3-6d26-4302-b31e-a0427d0895d7\" (UID: \"93fbc9a3-6d26-4302-b31e-a0427d0895d7\") " Sep 11 20:18:13.491691 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:13.491561 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cgn5m\" (UniqueName: \"kubernetes.io/projected/93fbc9a3-6d26-4302-b31e-a0427d0895d7-kube-api-access-cgn5m\") pod \"93fbc9a3-6d26-4302-b31e-a0427d0895d7\" (UID: \"93fbc9a3-6d26-4302-b31e-a0427d0895d7\") " Sep 11 20:18:13.491951 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:13.491919 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93fbc9a3-6d26-4302-b31e-a0427d0895d7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "93fbc9a3-6d26-4302-b31e-a0427d0895d7" (UID: "93fbc9a3-6d26-4302-b31e-a0427d0895d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:18:13.491951 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:13.491919 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c889f806-42e1-418c-8b3b-054e4f0b1250-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c889f806-42e1-418c-8b3b-054e4f0b1250" (UID: "c889f806-42e1-418c-8b3b-054e4f0b1250"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:18:13.493640 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:13.493614 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c889f806-42e1-418c-8b3b-054e4f0b1250-kube-api-access-9d9kt" (OuterVolumeSpecName: "kube-api-access-9d9kt") pod "c889f806-42e1-418c-8b3b-054e4f0b1250" (UID: "c889f806-42e1-418c-8b3b-054e4f0b1250"). InnerVolumeSpecName "kube-api-access-9d9kt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:18:13.493696 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:13.493655 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93fbc9a3-6d26-4302-b31e-a0427d0895d7-kube-api-access-cgn5m" (OuterVolumeSpecName: "kube-api-access-cgn5m") pod "93fbc9a3-6d26-4302-b31e-a0427d0895d7" (UID: "93fbc9a3-6d26-4302-b31e-a0427d0895d7"). InnerVolumeSpecName "kube-api-access-cgn5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:18:13.592561 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:13.592536 2546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93fbc9a3-6d26-4302-b31e-a0427d0895d7-plugins-config-volume\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:18:13.592561 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:13.592560 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgn5m\" (UniqueName: \"kubernetes.io/projected/93fbc9a3-6d26-4302-b31e-a0427d0895d7-kube-api-access-cgn5m\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:18:13.592761 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:13.592570 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d9kt\" (UniqueName: \"kubernetes.io/projected/c889f806-42e1-418c-8b3b-054e4f0b1250-kube-api-access-9d9kt\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:18:13.592761 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:13.592605 2546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c889f806-42e1-418c-8b3b-054e4f0b1250-plugins-config-volume\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:18:13.595220 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:13.595196 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:13.721943 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:13.721909 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:13.939712 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:13.939632 2546 generic.go:356] "Generic (PLEG): container finished" podID="93fbc9a3-6d26-4302-b31e-a0427d0895d7" containerID="ef74047c1d99f6e10cb24a0979cb58a6114e547992d96e4057d608bed31b32ee" exitCode=0 Sep 11 20:18:13.939868 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:13.939771 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-3c2adb-6685c45545-vft65" Sep 11 20:18:13.939868 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:13.939770 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-3c2adb-6685c45545-vft65" event={"ID":"93fbc9a3-6d26-4302-b31e-a0427d0895d7","Type":"ContainerDied","Data":"ef74047c1d99f6e10cb24a0979cb58a6114e547992d96e4057d608bed31b32ee"} Sep 11 20:18:13.939995 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:13.939942 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-3c2adb-6685c45545-vft65" event={"ID":"93fbc9a3-6d26-4302-b31e-a0427d0895d7","Type":"ContainerDied","Data":"fb3fbff3a852c528dd23b1dfbb3d9b9c7e89d2296d726cfcb3a027cd6431c5d4"} Sep 11 20:18:13.939995 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:13.939974 2546 scope.go:117] "RemoveContainer" containerID="ef74047c1d99f6e10cb24a0979cb58a6114e547992d96e4057d608bed31b32ee" Sep 11 20:18:13.941268 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:13.941244 2546 generic.go:356] "Generic (PLEG): container finished" podID="c889f806-42e1-418c-8b3b-054e4f0b1250" containerID="11a771ad2d58248960a70e02ceb784322917697afe54a62c0b045b7eb7dfcd5e" exitCode=0 Sep 11 20:18:13.941377 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:13.941283 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3c2adb-5d9448cfc8-7qrs4" event={"ID":"c889f806-42e1-418c-8b3b-054e4f0b1250","Type":"ContainerDied","Data":"11a771ad2d58248960a70e02ceb784322917697afe54a62c0b045b7eb7dfcd5e"} Sep 11 20:18:13.941377 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:13.941299 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3c2adb-5d9448cfc8-7qrs4" Sep 11 20:18:13.941377 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:13.941316 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3c2adb-5d9448cfc8-7qrs4" event={"ID":"c889f806-42e1-418c-8b3b-054e4f0b1250","Type":"ContainerDied","Data":"6c330cf0489fb75a08bb322321ec2b07a8ec328d43b9fdbead1f2aa82b0cab27"} Sep 11 20:18:13.955176 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:13.955138 2546 scope.go:117] "RemoveContainer" containerID="ef74047c1d99f6e10cb24a0979cb58a6114e547992d96e4057d608bed31b32ee" Sep 11 20:18:13.956254 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:18:13.955594 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef74047c1d99f6e10cb24a0979cb58a6114e547992d96e4057d608bed31b32ee\": container with ID starting with ef74047c1d99f6e10cb24a0979cb58a6114e547992d96e4057d608bed31b32ee not found: ID does not exist" containerID="ef74047c1d99f6e10cb24a0979cb58a6114e547992d96e4057d608bed31b32ee" Sep 11 20:18:13.956254 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:13.955640 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef74047c1d99f6e10cb24a0979cb58a6114e547992d96e4057d608bed31b32ee"} err="failed to get container status \"ef74047c1d99f6e10cb24a0979cb58a6114e547992d96e4057d608bed31b32ee\": rpc error: code = NotFound desc = could not find container \"ef74047c1d99f6e10cb24a0979cb58a6114e547992d96e4057d608bed31b32ee\": container with ID starting with ef74047c1d99f6e10cb24a0979cb58a6114e547992d96e4057d608bed31b32ee not found: ID does not exist" Sep 11 20:18:13.956254 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:13.955664 2546 scope.go:117] "RemoveContainer" containerID="11a771ad2d58248960a70e02ceb784322917697afe54a62c0b045b7eb7dfcd5e" Sep 11 20:18:13.972257 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:13.972170 2546 scope.go:117] "RemoveContainer" containerID="11a771ad2d58248960a70e02ceb784322917697afe54a62c0b045b7eb7dfcd5e" Sep 11 20:18:13.972671 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:18:13.972645 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11a771ad2d58248960a70e02ceb784322917697afe54a62c0b045b7eb7dfcd5e\": container with ID starting with 11a771ad2d58248960a70e02ceb784322917697afe54a62c0b045b7eb7dfcd5e not found: ID does not exist" containerID="11a771ad2d58248960a70e02ceb784322917697afe54a62c0b045b7eb7dfcd5e" Sep 11 20:18:13.972747 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:13.972681 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11a771ad2d58248960a70e02ceb784322917697afe54a62c0b045b7eb7dfcd5e"} err="failed to get container status \"11a771ad2d58248960a70e02ceb784322917697afe54a62c0b045b7eb7dfcd5e\": rpc error: code = NotFound desc = could not find container \"11a771ad2d58248960a70e02ceb784322917697afe54a62c0b045b7eb7dfcd5e\": container with ID starting with 11a771ad2d58248960a70e02ceb784322917697afe54a62c0b045b7eb7dfcd5e not found: ID does not exist" Sep 11 20:18:13.978403 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:13.978379 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-3c2adb-6685c45545-vft65"] Sep 11 20:18:13.980907 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:13.980885 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-3c2adb-6685c45545-vft65"] Sep 11 20:18:13.991436 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:13.991414 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-3c2adb-5d9448cfc8-7qrs4"] Sep 11 20:18:13.995201 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:13.995183 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-3c2adb-5d9448cfc8-7qrs4"] Sep 11 20:18:14.305110 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:14.305015 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-explicit-3c2adb-6685c45545-vft65" podUID="93fbc9a3-6d26-4302-b31e-a0427d0895d7" containerName="payload-processing" probeResult="failure" output="dial tcp 10.133.0.61:9004: i/o timeout" Sep 11 20:18:14.452652 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:14.452621 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93fbc9a3-6d26-4302-b31e-a0427d0895d7" path="/var/lib/kubelet/pods/93fbc9a3-6d26-4302-b31e-a0427d0895d7/volumes" Sep 11 20:18:14.453023 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:14.453010 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c889f806-42e1-418c-8b3b-054e4f0b1250" path="/var/lib/kubelet/pods/c889f806-42e1-418c-8b3b-054e4f0b1250/volumes" Sep 11 20:18:14.472185 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:14.472157 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:15.313167 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:15.313124 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:16.123721 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:16.123694 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:16.748891 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:16.748841 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:16.851922 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:16.851890 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:17.185300 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:17.185272 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:17.620606 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:17.620562 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:17.924133 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:17.924012 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:18.340827 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:18.340797 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:19.983110 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:19.983077 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:20.294772 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:20.294689 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:20.396147 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:20.396106 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:20.540167 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:20.540132 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:20.645809 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:20.645773 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:20.746222 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:20.746193 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:20.856980 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:20.856937 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:20.972428 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:20.972338 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:21.251463 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:21.251384 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs"] Sep 11 20:18:21.252080 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:21.252059 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d86ac50-085a-47a7-94aa-95ecbee3bd3c" containerName="payload-pre-processing" Sep 11 20:18:21.252170 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:21.252083 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d86ac50-085a-47a7-94aa-95ecbee3bd3c" containerName="payload-pre-processing" Sep 11 20:18:21.252170 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:21.252114 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="642e7b32-0972-49c8-8b03-9798bb7d0838" containerName="payload-processing" Sep 11 20:18:21.252170 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:21.252123 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="642e7b32-0972-49c8-8b03-9798bb7d0838" containerName="payload-processing" Sep 11 20:18:21.252170 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:21.252140 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93fbc9a3-6d26-4302-b31e-a0427d0895d7" containerName="payload-processing" Sep 11 20:18:21.252170 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:21.252148 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="93fbc9a3-6d26-4302-b31e-a0427d0895d7" containerName="payload-processing" Sep 11 20:18:21.252170 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:21.252163 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c889f806-42e1-418c-8b3b-054e4f0b1250" containerName="payload-pre-processing" Sep 11 20:18:21.252457 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:21.252173 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="c889f806-42e1-418c-8b3b-054e4f0b1250" containerName="payload-pre-processing" Sep 11 20:18:21.252457 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:21.252281 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="93fbc9a3-6d26-4302-b31e-a0427d0895d7" containerName="payload-processing" Sep 11 20:18:21.252457 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:21.252298 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="642e7b32-0972-49c8-8b03-9798bb7d0838" containerName="payload-processing" Sep 11 20:18:21.252457 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:21.252310 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d86ac50-085a-47a7-94aa-95ecbee3bd3c" containerName="payload-pre-processing" Sep 11 20:18:21.252457 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:21.252321 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="c889f806-42e1-418c-8b3b-054e4f0b1250" containerName="payload-pre-processing" Sep 11 20:18:21.256962 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:21.256938 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs" Sep 11 20:18:21.259334 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:21.259310 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-279dba-openshift-default-dockercfg-vjb9p\"" Sep 11 20:18:21.270426 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:21.270406 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs"] Sep 11 20:18:21.304998 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:21.304965 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:21.373482 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:21.373446 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5-istiod-ca-cert\") pod \"e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs\" (UID: \"1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5\") " pod="openshift-ingress/e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs" Sep 11 20:18:21.373690 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:21.373566 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5-istio-podinfo\") pod \"e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs\" (UID: \"1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5\") " pod="openshift-ingress/e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs" Sep 11 20:18:21.373690 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:21.373682 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5-workload-socket\") pod \"e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs\" (UID: \"1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5\") " pod="openshift-ingress/e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs" Sep 11 20:18:21.373827 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:21.373777 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5-workload-certs\") pod \"e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs\" (UID: \"1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5\") " pod="openshift-ingress/e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs" Sep 11 20:18:21.373827 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:21.373806 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5-istio-envoy\") pod \"e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs\" (UID: \"1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5\") " pod="openshift-ingress/e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs" Sep 11 20:18:21.373932 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:21.373846 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5-istio-token\") pod \"e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs\" (UID: \"1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5\") " pod="openshift-ingress/e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs" Sep 11 20:18:21.373932 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:21.373878 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6bfp\" (UniqueName: \"kubernetes.io/projected/1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5-kube-api-access-f6bfp\") pod \"e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs\" (UID: \"1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5\") " pod="openshift-ingress/e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs" Sep 11 20:18:21.373932 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:21.373926 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5-istio-data\") pod \"e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs\" (UID: \"1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5\") " pod="openshift-ingress/e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs" Sep 11 20:18:21.374086 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:21.374006 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5-credential-socket\") pod \"e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs\" (UID: \"1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5\") " pod="openshift-ingress/e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs" Sep 11 20:18:21.475506 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:21.475471 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5-istio-podinfo\") pod \"e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs\" (UID: \"1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5\") " pod="openshift-ingress/e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs" Sep 11 20:18:21.475506 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:21.475511 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5-workload-socket\") pod \"e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs\" (UID: \"1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5\") " pod="openshift-ingress/e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs" Sep 11 20:18:21.475782 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:21.475560 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5-workload-certs\") pod \"e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs\" (UID: \"1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5\") " pod="openshift-ingress/e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs" Sep 11 20:18:21.475782 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:21.475602 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5-istio-envoy\") pod \"e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs\" (UID: \"1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5\") " pod="openshift-ingress/e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs" Sep 11 20:18:21.475782 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:21.475626 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5-istio-token\") pod \"e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs\" (UID: \"1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5\") " pod="openshift-ingress/e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs" Sep 11 20:18:21.475782 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:21.475652 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6bfp\" (UniqueName: \"kubernetes.io/projected/1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5-kube-api-access-f6bfp\") pod \"e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs\" (UID: \"1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5\") " pod="openshift-ingress/e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs" Sep 11 20:18:21.475782 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:21.475684 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5-istio-data\") pod \"e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs\" (UID: \"1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5\") " pod="openshift-ingress/e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs" Sep 11 20:18:21.476036 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:21.475945 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5-credential-socket\") pod \"e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs\" (UID: \"1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5\") " pod="openshift-ingress/e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs" Sep 11 20:18:21.476096 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:21.476051 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5-istiod-ca-cert\") pod \"e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs\" (UID: \"1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5\") " pod="openshift-ingress/e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs" Sep 11 20:18:21.476152 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:21.476097 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5-workload-socket\") pod \"e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs\" (UID: \"1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5\") " pod="openshift-ingress/e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs" Sep 11 20:18:21.476296 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:21.476269 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5-workload-certs\") pod \"e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs\" (UID: \"1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5\") " pod="openshift-ingress/e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs" Sep 11 20:18:21.476435 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:21.476265 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5-istio-data\") pod \"e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs\" (UID: \"1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5\") " pod="openshift-ingress/e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs" Sep 11 20:18:21.476668 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:21.476649 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5-credential-socket\") pod \"e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs\" (UID: \"1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5\") " pod="openshift-ingress/e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs" Sep 11 20:18:21.477149 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:21.477122 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5-istiod-ca-cert\") pod \"e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs\" (UID: \"1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5\") " pod="openshift-ingress/e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs" Sep 11 20:18:21.478256 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:21.478236 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5-istio-envoy\") pod \"e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs\" (UID: \"1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5\") " pod="openshift-ingress/e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs" Sep 11 20:18:21.478373 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:21.478344 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5-istio-podinfo\") pod \"e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs\" (UID: \"1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5\") " pod="openshift-ingress/e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs" Sep 11 20:18:21.486981 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:21.486960 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6bfp\" (UniqueName: \"kubernetes.io/projected/1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5-kube-api-access-f6bfp\") pod \"e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs\" (UID: \"1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5\") " pod="openshift-ingress/e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs" Sep 11 20:18:21.487074 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:21.486967 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5-istio-token\") pod \"e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs\" (UID: \"1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5\") " pod="openshift-ingress/e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs" Sep 11 20:18:21.513125 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:21.513055 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:21.567833 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:21.567804 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs" Sep 11 20:18:21.618620 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:21.617738 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:21.868884 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:21.868856 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs"] Sep 11 20:18:21.870287 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:18:21.870253 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c1e9a89_57d7_4bf2_aa27_ebbea5f86dd5.slice/crio-de6114e56300d14242b77715d64162721d5c89f26c5560279b89ba455278a3b7 WatchSource:0}: Error finding container de6114e56300d14242b77715d64162721d5c89f26c5560279b89ba455278a3b7: Status 404 returned error can't find the container with id de6114e56300d14242b77715d64162721d5c89f26c5560279b89ba455278a3b7 Sep 11 20:18:21.872721 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:21.872685 2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 20:18:21.872830 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:21.872748 2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 20:18:21.872830 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:21.872784 2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 20:18:21.892771 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:21.892738 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:21.994209 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:21.994176 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs" event={"ID":"1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5","Type":"ContainerStarted","Data":"9055e7b7f97b3c323af75d55c946da400516a5e0a2950196cf92d9c0b81b2b86"} Sep 11 20:18:21.994480 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:21.994453 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs" event={"ID":"1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5","Type":"ContainerStarted","Data":"de6114e56300d14242b77715d64162721d5c89f26c5560279b89ba455278a3b7"} Sep 11 20:18:22.015858 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:22.015807 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs" podStartSLOduration=1.015788264 podStartE2EDuration="1.015788264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:18:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:18:22.013798966 +0000 UTC m=+1072.061849311" watchObservedRunningTime="2026-09-11 20:18:22.015788264 +0000 UTC m=+1072.063838611" Sep 11 20:18:22.290942 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:22.290857 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:22.389125 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:22.389091 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:22.492413 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:22.492382 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:22.568550 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:22.568471 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs" Sep 11 20:18:22.575155 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:22.575131 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs" Sep 11 20:18:22.842309 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:22.842279 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:23.000077 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:23.000050 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs" Sep 11 20:18:23.000995 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:23.000970 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs" Sep 11 20:18:23.548284 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:23.548257 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:24.297791 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:24.297761 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ea327d-7c65697774-l6rzd"] Sep 11 20:18:24.301692 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:24.301668 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ea327d-7c65697774-l6rzd" Sep 11 20:18:24.304643 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:24.304622 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-ea327d-serving-cert\"" Sep 11 20:18:24.310912 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:24.310792 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ea327d-7c65697774-l6rzd"] Sep 11 20:18:24.335477 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:24.335445 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:24.411046 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:24.411010 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc977a98-1ebc-44cf-8eb0-6f6693b092d3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-ea327d-7c65697774-l6rzd\" (UID: \"dc977a98-1ebc-44cf-8eb0-6f6693b092d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ea327d-7c65697774-l6rzd" Sep 11 20:18:24.411213 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:24.411084 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc977a98-1ebc-44cf-8eb0-6f6693b092d3-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-ea327d-7c65697774-l6rzd\" (UID: \"dc977a98-1ebc-44cf-8eb0-6f6693b092d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ea327d-7c65697774-l6rzd" Sep 11 20:18:24.411213 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:24.411123 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc977a98-1ebc-44cf-8eb0-6f6693b092d3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-ea327d-7c65697774-l6rzd\" (UID: \"dc977a98-1ebc-44cf-8eb0-6f6693b092d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ea327d-7c65697774-l6rzd" Sep 11 20:18:24.411213 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:24.411173 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkjtx\" (UniqueName: \"kubernetes.io/projected/dc977a98-1ebc-44cf-8eb0-6f6693b092d3-kube-api-access-fkjtx\") pod \"maas-api-e2e-shared-a-w2-ea327d-7c65697774-l6rzd\" (UID: \"dc977a98-1ebc-44cf-8eb0-6f6693b092d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ea327d-7c65697774-l6rzd" Sep 11 20:18:24.439096 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:24.439062 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:24.512852 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:24.512822 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc977a98-1ebc-44cf-8eb0-6f6693b092d3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-ea327d-7c65697774-l6rzd\" (UID: \"dc977a98-1ebc-44cf-8eb0-6f6693b092d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ea327d-7c65697774-l6rzd" Sep 11 20:18:24.513016 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:24.512874 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc977a98-1ebc-44cf-8eb0-6f6693b092d3-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-ea327d-7c65697774-l6rzd\" (UID: \"dc977a98-1ebc-44cf-8eb0-6f6693b092d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ea327d-7c65697774-l6rzd" Sep 11 20:18:24.513016 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:24.512912 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc977a98-1ebc-44cf-8eb0-6f6693b092d3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-ea327d-7c65697774-l6rzd\" (UID: \"dc977a98-1ebc-44cf-8eb0-6f6693b092d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ea327d-7c65697774-l6rzd" Sep 11 20:18:24.513016 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:24.512958 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkjtx\" (UniqueName: \"kubernetes.io/projected/dc977a98-1ebc-44cf-8eb0-6f6693b092d3-kube-api-access-fkjtx\") pod \"maas-api-e2e-shared-a-w2-ea327d-7c65697774-l6rzd\" (UID: \"dc977a98-1ebc-44cf-8eb0-6f6693b092d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ea327d-7c65697774-l6rzd" Sep 11 20:18:24.513633 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:24.513610 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc977a98-1ebc-44cf-8eb0-6f6693b092d3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-ea327d-7c65697774-l6rzd\" (UID: \"dc977a98-1ebc-44cf-8eb0-6f6693b092d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ea327d-7c65697774-l6rzd" Sep 11 20:18:24.515284 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:24.515264 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc977a98-1ebc-44cf-8eb0-6f6693b092d3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-ea327d-7c65697774-l6rzd\" (UID: \"dc977a98-1ebc-44cf-8eb0-6f6693b092d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ea327d-7c65697774-l6rzd" Sep 11 20:18:24.515381 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:24.515363 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc977a98-1ebc-44cf-8eb0-6f6693b092d3-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-ea327d-7c65697774-l6rzd\" (UID: \"dc977a98-1ebc-44cf-8eb0-6f6693b092d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ea327d-7c65697774-l6rzd" Sep 11 20:18:24.521694 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:24.521674 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkjtx\" (UniqueName: \"kubernetes.io/projected/dc977a98-1ebc-44cf-8eb0-6f6693b092d3-kube-api-access-fkjtx\") pod \"maas-api-e2e-shared-a-w2-ea327d-7c65697774-l6rzd\" (UID: \"dc977a98-1ebc-44cf-8eb0-6f6693b092d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ea327d-7c65697774-l6rzd" Sep 11 20:18:24.543449 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:24.543424 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:24.617122 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:24.617099 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ea327d-7c65697774-l6rzd" Sep 11 20:18:24.850394 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:24.850359 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:24.894690 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:24.894667 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ea327d-7c65697774-l6rzd"] Sep 11 20:18:24.897283 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:18:24.897257 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc977a98_1ebc_44cf_8eb0_6f6693b092d3.slice/crio-c5e170179705069af29006aa02c044614167cb21365575c4e30b1b5d4a205c06 WatchSource:0}: Error finding container c5e170179705069af29006aa02c044614167cb21365575c4e30b1b5d4a205c06: Status 404 returned error can't find the container with id c5e170179705069af29006aa02c044614167cb21365575c4e30b1b5d4a205c06 Sep 11 20:18:24.899317 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:24.899297 2546 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 20:18:24.929018 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:24.928991 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:24.933684 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:24.933664 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ea327d-6f44c946c8-ccphp"] Sep 11 20:18:24.938554 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:24.938537 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ea327d-6f44c946c8-ccphp" Sep 11 20:18:24.940694 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:24.940677 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-ea327d\"" Sep 11 20:18:24.940807 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:24.940732 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-ea327d-dockercfg-2q2fc\"" Sep 11 20:18:24.949657 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:24.949638 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ea327d-6f44c946c8-ccphp"] Sep 11 20:18:24.970404 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:24.970379 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-ea327d-7999bdbc7-xpxc5"] Sep 11 20:18:24.974075 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:24.974058 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ea327d-7999bdbc7-xpxc5" Sep 11 20:18:24.982805 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:24.982783 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-ea327d-7999bdbc7-xpxc5"] Sep 11 20:18:25.011444 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:25.011414 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ea327d-7c65697774-l6rzd" event={"ID":"dc977a98-1ebc-44cf-8eb0-6f6693b092d3","Type":"ContainerStarted","Data":"c5e170179705069af29006aa02c044614167cb21365575c4e30b1b5d4a205c06"} Sep 11 20:18:25.018642 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:25.018621 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3abcbcd7-5ecd-417c-a2c2-378966ad14fd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-ea327d-7999bdbc7-xpxc5\" (UID: \"3abcbcd7-5ecd-417c-a2c2-378966ad14fd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ea327d-7999bdbc7-xpxc5" Sep 11 20:18:25.018744 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:25.018660 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7301644-7f5a-4fb0-a7f3-908f028c938c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-ea327d-6f44c946c8-ccphp\" (UID: \"b7301644-7f5a-4fb0-a7f3-908f028c938c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ea327d-6f44c946c8-ccphp" Sep 11 20:18:25.018744 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:25.018685 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g9vk\" (UniqueName: \"kubernetes.io/projected/3abcbcd7-5ecd-417c-a2c2-378966ad14fd-kube-api-access-9g9vk\") pod \"payload-processing-e2e-shared-a-w2-ea327d-7999bdbc7-xpxc5\" (UID: \"3abcbcd7-5ecd-417c-a2c2-378966ad14fd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ea327d-7999bdbc7-xpxc5" Sep 11 20:18:25.018858 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:25.018789 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8bc4\" (UniqueName: \"kubernetes.io/projected/b7301644-7f5a-4fb0-a7f3-908f028c938c-kube-api-access-v8bc4\") pod \"payload-pre-processing-e2e-shared-a-w2-ea327d-6f44c946c8-ccphp\" (UID: \"b7301644-7f5a-4fb0-a7f3-908f028c938c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ea327d-6f44c946c8-ccphp" Sep 11 20:18:25.120162 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:25.120126 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8bc4\" (UniqueName: \"kubernetes.io/projected/b7301644-7f5a-4fb0-a7f3-908f028c938c-kube-api-access-v8bc4\") pod \"payload-pre-processing-e2e-shared-a-w2-ea327d-6f44c946c8-ccphp\" (UID: \"b7301644-7f5a-4fb0-a7f3-908f028c938c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ea327d-6f44c946c8-ccphp" Sep 11 20:18:25.120365 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:25.120250 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3abcbcd7-5ecd-417c-a2c2-378966ad14fd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-ea327d-7999bdbc7-xpxc5\" (UID: \"3abcbcd7-5ecd-417c-a2c2-378966ad14fd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ea327d-7999bdbc7-xpxc5" Sep 11 20:18:25.120450 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:25.120431 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7301644-7f5a-4fb0-a7f3-908f028c938c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-ea327d-6f44c946c8-ccphp\" (UID: \"b7301644-7f5a-4fb0-a7f3-908f028c938c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ea327d-6f44c946c8-ccphp" Sep 11 20:18:25.120534 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:25.120511 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g9vk\" (UniqueName: \"kubernetes.io/projected/3abcbcd7-5ecd-417c-a2c2-378966ad14fd-kube-api-access-9g9vk\") pod \"payload-processing-e2e-shared-a-w2-ea327d-7999bdbc7-xpxc5\" (UID: \"3abcbcd7-5ecd-417c-a2c2-378966ad14fd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ea327d-7999bdbc7-xpxc5" Sep 11 20:18:25.120993 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:25.120973 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3abcbcd7-5ecd-417c-a2c2-378966ad14fd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-ea327d-7999bdbc7-xpxc5\" (UID: \"3abcbcd7-5ecd-417c-a2c2-378966ad14fd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ea327d-7999bdbc7-xpxc5" Sep 11 20:18:25.121091 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:25.121074 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7301644-7f5a-4fb0-a7f3-908f028c938c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-ea327d-6f44c946c8-ccphp\" (UID: \"b7301644-7f5a-4fb0-a7f3-908f028c938c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ea327d-6f44c946c8-ccphp" Sep 11 20:18:25.127848 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:25.127821 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8bc4\" (UniqueName: \"kubernetes.io/projected/b7301644-7f5a-4fb0-a7f3-908f028c938c-kube-api-access-v8bc4\") pod \"payload-pre-processing-e2e-shared-a-w2-ea327d-6f44c946c8-ccphp\" (UID: \"b7301644-7f5a-4fb0-a7f3-908f028c938c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ea327d-6f44c946c8-ccphp" Sep 11 20:18:25.128089 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:25.128069 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g9vk\" (UniqueName: \"kubernetes.io/projected/3abcbcd7-5ecd-417c-a2c2-378966ad14fd-kube-api-access-9g9vk\") pod \"payload-processing-e2e-shared-a-w2-ea327d-7999bdbc7-xpxc5\" (UID: \"3abcbcd7-5ecd-417c-a2c2-378966ad14fd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ea327d-7999bdbc7-xpxc5" Sep 11 20:18:25.250819 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:25.250744 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ea327d-6f44c946c8-ccphp" Sep 11 20:18:25.268672 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:25.268644 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:25.286337 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:25.286293 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ea327d-7999bdbc7-xpxc5" Sep 11 20:18:25.528346 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:25.528306 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ea327d-6f44c946c8-ccphp"] Sep 11 20:18:25.529513 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:18:25.529449 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7301644_7f5a_4fb0_a7f3_908f028c938c.slice/crio-a7bd9cd007976cce8c3e61bb2e45ad0fe96bf49c132086cb10782fcdf0839946 WatchSource:0}: Error finding container a7bd9cd007976cce8c3e61bb2e45ad0fe96bf49c132086cb10782fcdf0839946: Status 404 returned error can't find the container with id a7bd9cd007976cce8c3e61bb2e45ad0fe96bf49c132086cb10782fcdf0839946 Sep 11 20:18:25.570250 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:25.570225 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-ea327d-7999bdbc7-xpxc5"] Sep 11 20:18:25.572462 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:18:25.572437 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3abcbcd7_5ecd_417c_a2c2_378966ad14fd.slice/crio-cb104e87fb3438d69c967c34abe1704f9640c283748844ad037783cf63d2bc1e WatchSource:0}: Error finding container cb104e87fb3438d69c967c34abe1704f9640c283748844ad037783cf63d2bc1e: Status 404 returned error can't find the container with id cb104e87fb3438d69c967c34abe1704f9640c283748844ad037783cf63d2bc1e Sep 11 20:18:26.018684 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:26.018646 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ea327d-6f44c946c8-ccphp" event={"ID":"b7301644-7f5a-4fb0-a7f3-908f028c938c","Type":"ContainerStarted","Data":"a521b6048cebb7aba1b156874051b0937749211ab0857a18e2b50bb8fefb17ff"} Sep 11 20:18:26.019108 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:26.018702 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ea327d-6f44c946c8-ccphp" event={"ID":"b7301644-7f5a-4fb0-a7f3-908f028c938c","Type":"ContainerStarted","Data":"a7bd9cd007976cce8c3e61bb2e45ad0fe96bf49c132086cb10782fcdf0839946"} Sep 11 20:18:26.019108 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:26.018815 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ea327d-6f44c946c8-ccphp" Sep 11 20:18:26.020431 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:26.020404 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ea327d-7999bdbc7-xpxc5" event={"ID":"3abcbcd7-5ecd-417c-a2c2-378966ad14fd","Type":"ContainerStarted","Data":"3b13f7e96f114cc39a97e2260ec0622489092095d6cffade34c8d45644e3ea61"} Sep 11 20:18:26.020607 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:26.020457 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ea327d-7999bdbc7-xpxc5" Sep 11 20:18:26.020607 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:26.020481 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ea327d-7999bdbc7-xpxc5" event={"ID":"3abcbcd7-5ecd-417c-a2c2-378966ad14fd","Type":"ContainerStarted","Data":"cb104e87fb3438d69c967c34abe1704f9640c283748844ad037783cf63d2bc1e"} Sep 11 20:18:26.022001 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:26.021978 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ea327d-7c65697774-l6rzd" event={"ID":"dc977a98-1ebc-44cf-8eb0-6f6693b092d3","Type":"ContainerStarted","Data":"35a82174331b01a8f1c06cade1411877cf5426a6cfd8967901ff8e38f39b29ee"} Sep 11 20:18:26.022150 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:26.022134 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ea327d-7c65697774-l6rzd" Sep 11 20:18:26.037078 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:26.037029 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ea327d-6f44c946c8-ccphp" podStartSLOduration=2.037011586 podStartE2EDuration="2.037011586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:18:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:18:26.034683355 +0000 UTC m=+1076.082733698" watchObservedRunningTime="2026-09-11 20:18:26.037011586 +0000 UTC m=+1076.085061934" Sep 11 20:18:26.040121 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:26.040094 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:26.054228 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:26.054188 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ea327d-7c65697774-l6rzd" podStartSLOduration=1.5882315500000002 podStartE2EDuration="2.054175957s" totalImagesPullingTime="465.944407ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:18:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:18:26.051441708 +0000 UTC m=+1076.099492053" watchObservedRunningTime="2026-09-11 20:18:26.054175957 +0000 UTC m=+1076.102226334" Sep 11 20:18:26.070544 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:26.070497 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ea327d-7999bdbc7-xpxc5" podStartSLOduration=2.070486181 podStartE2EDuration="2.070486181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:18:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:18:26.068598424 +0000 UTC m=+1076.116648771" watchObservedRunningTime="2026-09-11 20:18:26.070486181 +0000 UTC m=+1076.118536561" Sep 11 20:18:26.789796 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:26.789769 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:27.480326 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:27.480299 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:32.034391 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:32.034363 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ea327d-7c65697774-l6rzd" Sep 11 20:18:37.030914 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:37.030879 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ea327d-6f44c946c8-ccphp" Sep 11 20:18:37.031351 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:37.031063 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ea327d-7999bdbc7-xpxc5" Sep 11 20:18:45.214189 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:45.214153 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:46.484754 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:46.484715 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:47.334819 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:47.334785 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:48.016180 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:48.016150 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:48.800694 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:48.800659 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:49.613991 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:49.613958 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:50.307447 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:50.307414 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:50.529810 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:50.529781 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:50.840901 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:50.840868 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:50.976429 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:50.976394 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:51.133803 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:51.133728 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:51.233925 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:51.233898 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:51.332940 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:51.332909 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:53.013097 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:53.013068 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:53.802731 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:53.802683 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:53.898452 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:53.898421 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:54.798330 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:54.798298 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:55.606953 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:55.606923 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:56.385037 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:56.384997 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:57.061606 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:57.061552 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:57.743488 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:57.743454 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:57.843916 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:57.843880 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:58.308727 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:58.308700 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:58.407729 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:58.407697 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:58.438076 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:58.438049 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c7e744-855cc7f695-nbv9z"] Sep 11 20:18:58.443269 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:58.443247 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c7e744-855cc7f695-nbv9z" Sep 11 20:18:58.445432 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:58.445409 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-c7e744\"" Sep 11 20:18:58.445561 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:58.445409 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-c7e744-dockercfg-r6k2r\"" Sep 11 20:18:58.459238 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:58.459208 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c7e744-855cc7f695-nbv9z"] Sep 11 20:18:58.530656 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:58.530615 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c7e744-6958dc9df4-cd6b2"] Sep 11 20:18:58.534441 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:58.534388 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c7e744-6958dc9df4-cd6b2" Sep 11 20:18:58.544410 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:58.544383 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9c72ca7-dd23-4606-b917-7c7df7de2818-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-c7e744-855cc7f695-nbv9z\" (UID: \"e9c72ca7-dd23-4606-b917-7c7df7de2818\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c7e744-855cc7f695-nbv9z" Sep 11 20:18:58.544548 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:58.544469 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqtwj\" (UniqueName: \"kubernetes.io/projected/e9c72ca7-dd23-4606-b917-7c7df7de2818-kube-api-access-nqtwj\") pod \"payload-processing-e2e-shared-b-w2-c7e744-855cc7f695-nbv9z\" (UID: \"e9c72ca7-dd23-4606-b917-7c7df7de2818\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c7e744-855cc7f695-nbv9z" Sep 11 20:18:58.565600 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:58.565410 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c7e744-6958dc9df4-cd6b2"] Sep 11 20:18:58.645793 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:58.645761 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74519b19-232e-4c39-905f-d43ab2f80a92-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-c7e744-6958dc9df4-cd6b2\" (UID: \"74519b19-232e-4c39-905f-d43ab2f80a92\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c7e744-6958dc9df4-cd6b2" Sep 11 20:18:58.645971 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:58.645875 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9c72ca7-dd23-4606-b917-7c7df7de2818-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-c7e744-855cc7f695-nbv9z\" (UID: \"e9c72ca7-dd23-4606-b917-7c7df7de2818\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c7e744-855cc7f695-nbv9z" Sep 11 20:18:58.645971 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:58.645923 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5fnx\" (UniqueName: \"kubernetes.io/projected/74519b19-232e-4c39-905f-d43ab2f80a92-kube-api-access-z5fnx\") pod \"payload-pre-processing-e2e-shared-b-w2-c7e744-6958dc9df4-cd6b2\" (UID: \"74519b19-232e-4c39-905f-d43ab2f80a92\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c7e744-6958dc9df4-cd6b2" Sep 11 20:18:58.646095 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:58.645985 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqtwj\" (UniqueName: \"kubernetes.io/projected/e9c72ca7-dd23-4606-b917-7c7df7de2818-kube-api-access-nqtwj\") pod \"payload-processing-e2e-shared-b-w2-c7e744-855cc7f695-nbv9z\" (UID: \"e9c72ca7-dd23-4606-b917-7c7df7de2818\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c7e744-855cc7f695-nbv9z" Sep 11 20:18:58.646627 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:58.646571 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9c72ca7-dd23-4606-b917-7c7df7de2818-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-c7e744-855cc7f695-nbv9z\" (UID: \"e9c72ca7-dd23-4606-b917-7c7df7de2818\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c7e744-855cc7f695-nbv9z" Sep 11 20:18:58.647029 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:58.647001 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:18:58.661091 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:58.661061 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqtwj\" (UniqueName: \"kubernetes.io/projected/e9c72ca7-dd23-4606-b917-7c7df7de2818-kube-api-access-nqtwj\") pod \"payload-processing-e2e-shared-b-w2-c7e744-855cc7f695-nbv9z\" (UID: \"e9c72ca7-dd23-4606-b917-7c7df7de2818\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c7e744-855cc7f695-nbv9z" Sep 11 20:18:58.747702 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:58.747667 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74519b19-232e-4c39-905f-d43ab2f80a92-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-c7e744-6958dc9df4-cd6b2\" (UID: \"74519b19-232e-4c39-905f-d43ab2f80a92\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c7e744-6958dc9df4-cd6b2" Sep 11 20:18:58.748080 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:58.747780 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5fnx\" (UniqueName: \"kubernetes.io/projected/74519b19-232e-4c39-905f-d43ab2f80a92-kube-api-access-z5fnx\") pod \"payload-pre-processing-e2e-shared-b-w2-c7e744-6958dc9df4-cd6b2\" (UID: \"74519b19-232e-4c39-905f-d43ab2f80a92\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c7e744-6958dc9df4-cd6b2" Sep 11 20:18:58.748426 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:58.748408 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74519b19-232e-4c39-905f-d43ab2f80a92-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-c7e744-6958dc9df4-cd6b2\" (UID: \"74519b19-232e-4c39-905f-d43ab2f80a92\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c7e744-6958dc9df4-cd6b2" Sep 11 20:18:58.756875 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:58.756850 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5fnx\" (UniqueName: \"kubernetes.io/projected/74519b19-232e-4c39-905f-d43ab2f80a92-kube-api-access-z5fnx\") pod \"payload-pre-processing-e2e-shared-b-w2-c7e744-6958dc9df4-cd6b2\" (UID: \"74519b19-232e-4c39-905f-d43ab2f80a92\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c7e744-6958dc9df4-cd6b2" Sep 11 20:18:58.762754 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:58.762736 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c7e744-855cc7f695-nbv9z" Sep 11 20:18:58.849597 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:58.849516 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c7e744-6958dc9df4-cd6b2" Sep 11 20:18:59.034412 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:59.034389 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c7e744-855cc7f695-nbv9z"] Sep 11 20:18:59.036801 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:18:59.036771 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9c72ca7_dd23_4606_b917_7c7df7de2818.slice/crio-1d0edc5330178883daf56285bda184c8e4a75c64bfcb14b0a34a11fe59346b38 WatchSource:0}: Error finding container 1d0edc5330178883daf56285bda184c8e4a75c64bfcb14b0a34a11fe59346b38: Status 404 returned error can't find the container with id 1d0edc5330178883daf56285bda184c8e4a75c64bfcb14b0a34a11fe59346b38 Sep 11 20:18:59.123546 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:59.123511 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c7e744-6958dc9df4-cd6b2"] Sep 11 20:18:59.127179 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:18:59.127150 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74519b19_232e_4c39_905f_d43ab2f80a92.slice/crio-d266d8ddeebd3bcf2fbce3850298b02cef759cbc1aac6756eb3b9e2b7f1b60a0 WatchSource:0}: Error finding container d266d8ddeebd3bcf2fbce3850298b02cef759cbc1aac6756eb3b9e2b7f1b60a0: Status 404 returned error can't find the container with id d266d8ddeebd3bcf2fbce3850298b02cef759cbc1aac6756eb3b9e2b7f1b60a0 Sep 11 20:18:59.227630 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:59.227327 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c7e744-6958dc9df4-cd6b2" event={"ID":"74519b19-232e-4c39-905f-d43ab2f80a92","Type":"ContainerStarted","Data":"03158ba7e2fce6ceb7d419bec08f9f98f4a1b42cba13c6229103e47c7c8d8221"} Sep 11 20:18:59.227630 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:59.227393 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c7e744-6958dc9df4-cd6b2" event={"ID":"74519b19-232e-4c39-905f-d43ab2f80a92","Type":"ContainerStarted","Data":"d266d8ddeebd3bcf2fbce3850298b02cef759cbc1aac6756eb3b9e2b7f1b60a0"} Sep 11 20:18:59.227993 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:59.227743 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c7e744-6958dc9df4-cd6b2" Sep 11 20:18:59.229756 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:59.229728 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c7e744-855cc7f695-nbv9z" event={"ID":"e9c72ca7-dd23-4606-b917-7c7df7de2818","Type":"ContainerStarted","Data":"d0c66203ca3e5a725f09fac8380879ba3a4aa25e3256bd6d3d5235506e321163"} Sep 11 20:18:59.229869 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:59.229782 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c7e744-855cc7f695-nbv9z" event={"ID":"e9c72ca7-dd23-4606-b917-7c7df7de2818","Type":"ContainerStarted","Data":"1d0edc5330178883daf56285bda184c8e4a75c64bfcb14b0a34a11fe59346b38"} Sep 11 20:18:59.229869 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:59.229813 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c7e744-855cc7f695-nbv9z" Sep 11 20:18:59.243681 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:59.243636 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c7e744-6958dc9df4-cd6b2" podStartSLOduration=1.243620282 podStartE2EDuration="1.243620282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:18:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:18:59.242859691 +0000 UTC m=+1109.290910034" watchObservedRunningTime="2026-09-11 20:18:59.243620282 +0000 UTC m=+1109.291670627" Sep 11 20:18:59.261468 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:18:59.261427 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c7e744-855cc7f695-nbv9z" podStartSLOduration=1.261411536 podStartE2EDuration="1.261411536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:18:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:18:59.259374267 +0000 UTC m=+1109.307424606" watchObservedRunningTime="2026-09-11 20:18:59.261411536 +0000 UTC m=+1109.309461933" Sep 11 20:19:08.707166 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:08.707081 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:19:10.236953 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:10.236924 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c7e744-6958dc9df4-cd6b2" Sep 11 20:19:10.237315 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:10.237105 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c7e744-855cc7f695-nbv9z" Sep 11 20:19:19.973210 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:19.973175 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:19:20.708487 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:20.708456 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:19:21.427134 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:21.427086 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:19:22.109821 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:22.109769 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:19:22.897378 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:22.897346 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:19:23.689511 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:23.689481 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:19:23.788572 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:23.788541 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:19:24.468633 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:24.468601 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:19:32.466452 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:32.466421 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:19:32.763369 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:32.763291 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:19:33.318585 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:33.318541 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:19:35.237185 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:35.237125 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:19:35.962983 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:35.962942 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:19:36.053747 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:36.053719 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:19:36.876832 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:36.876788 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:19:37.688269 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:37.688235 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:19:38.531258 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:38.531226 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:19:39.222669 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:39.222637 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:19:39.907367 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:39.907336 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:19:40.731709 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:40.731679 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:19:40.832160 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:40.832125 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:19:41.186437 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:41.186408 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:19:41.285555 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:41.285521 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:19:41.519636 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:41.519556 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:19:46.128000 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:46.127955 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:19:47.027098 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:47.027066 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:19:47.927670 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:47.927635 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:19:48.623267 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:48.623237 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:19:49.407936 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:49.407890 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:19:50.070181 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:50.070153 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:19:50.764795 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:50.764758 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:19:51.719300 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:51.719270 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:19:52.408765 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:52.408737 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:19:53.266809 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:53.266726 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:19:53.918977 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:53.918945 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:19:54.522432 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:54.522379 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:19:54.833990 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:54.833915 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:19:55.540402 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:55.540369 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:19:55.785792 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:55.785759 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:19:56.356462 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:56.356430 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:19:57.114563 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:57.114533 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:19:57.784668 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:57.784632 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:19:58.348610 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:58.348053 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b3e1dd-545f764567-7qgrd"] Sep 11 20:19:58.348610 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:58.348308 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3e1dd-545f764567-7qgrd" podUID="42b48ad0-8671-4297-8af7-3b0cb3e6b87e" containerName="maas-api" containerID="cri-o://cecd0a66fc9565732da59084cbbeb41055e2b3962582acacaf4a68eec1aad5b9" gracePeriod=30 Sep 11 20:19:58.372105 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:58.372077 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:19:58.412811 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:58.412780 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b3e1dd-bff5699cd-dv665"] Sep 11 20:19:58.412990 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:58.412966 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b3e1dd-bff5699cd-dv665" podUID="188761e4-5b30-40cc-85ec-f4c9b77be511" containerName="payload-pre-processing" containerID="cri-o://8729ea5d0fc95ba31d3b171ac9dd28b6da203c8d16c57e792ea35c613559e32e" gracePeriod=30 Sep 11 20:19:58.469670 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:58.469642 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:19:58.575454 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:58.575417 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:19:58.590471 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:58.590408 2546 generic.go:356] "Generic (PLEG): container finished" podID="42b48ad0-8671-4297-8af7-3b0cb3e6b87e" containerID="cecd0a66fc9565732da59084cbbeb41055e2b3962582acacaf4a68eec1aad5b9" exitCode=0 Sep 11 20:19:58.590617 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:58.590478 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3e1dd-545f764567-7qgrd" event={"ID":"42b48ad0-8671-4297-8af7-3b0cb3e6b87e","Type":"ContainerDied","Data":"cecd0a66fc9565732da59084cbbeb41055e2b3962582acacaf4a68eec1aad5b9"} Sep 11 20:19:58.592370 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:58.592349 2546 generic.go:356] "Generic (PLEG): container finished" podID="188761e4-5b30-40cc-85ec-f4c9b77be511" containerID="8729ea5d0fc95ba31d3b171ac9dd28b6da203c8d16c57e792ea35c613559e32e" exitCode=0 Sep 11 20:19:58.592473 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:58.592415 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b3e1dd-bff5699cd-dv665" event={"ID":"188761e4-5b30-40cc-85ec-f4c9b77be511","Type":"ContainerDied","Data":"8729ea5d0fc95ba31d3b171ac9dd28b6da203c8d16c57e792ea35c613559e32e"} Sep 11 20:19:58.644274 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:58.644188 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:19:58.729265 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:58.729233 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3e1dd-545f764567-7qgrd" Sep 11 20:19:58.732720 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:58.732701 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b3e1dd-bff5699cd-dv665" Sep 11 20:19:58.752687 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:58.752420 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q9nkw\" (UniqueName: \"kubernetes.io/projected/188761e4-5b30-40cc-85ec-f4c9b77be511-kube-api-access-q9nkw\") pod \"188761e4-5b30-40cc-85ec-f4c9b77be511\" (UID: \"188761e4-5b30-40cc-85ec-f4c9b77be511\") " Sep 11 20:19:58.752687 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:58.752531 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ghx5x\" (UniqueName: \"kubernetes.io/projected/42b48ad0-8671-4297-8af7-3b0cb3e6b87e-kube-api-access-ghx5x\") pod \"42b48ad0-8671-4297-8af7-3b0cb3e6b87e\" (UID: \"42b48ad0-8671-4297-8af7-3b0cb3e6b87e\") " Sep 11 20:19:58.752687 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:58.752615 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42b48ad0-8671-4297-8af7-3b0cb3e6b87e-metrics-certs\") pod \"42b48ad0-8671-4297-8af7-3b0cb3e6b87e\" (UID: \"42b48ad0-8671-4297-8af7-3b0cb3e6b87e\") " Sep 11 20:19:58.753375 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:58.752818 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42b48ad0-8671-4297-8af7-3b0cb3e6b87e-maas-api-tls\") pod \"42b48ad0-8671-4297-8af7-3b0cb3e6b87e\" (UID: \"42b48ad0-8671-4297-8af7-3b0cb3e6b87e\") " Sep 11 20:19:58.753375 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:58.752971 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/188761e4-5b30-40cc-85ec-f4c9b77be511-plugins-config-volume\") pod \"188761e4-5b30-40cc-85ec-f4c9b77be511\" (UID: \"188761e4-5b30-40cc-85ec-f4c9b77be511\") " Sep 11 20:19:58.753375 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:58.753153 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42b48ad0-8671-4297-8af7-3b0cb3e6b87e-openshift-service-ca\") pod \"42b48ad0-8671-4297-8af7-3b0cb3e6b87e\" (UID: \"42b48ad0-8671-4297-8af7-3b0cb3e6b87e\") " Sep 11 20:19:58.754542 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:58.753919 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42b48ad0-8671-4297-8af7-3b0cb3e6b87e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "42b48ad0-8671-4297-8af7-3b0cb3e6b87e" (UID: "42b48ad0-8671-4297-8af7-3b0cb3e6b87e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:19:58.754542 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:58.754335 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/188761e4-5b30-40cc-85ec-f4c9b77be511-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "188761e4-5b30-40cc-85ec-f4c9b77be511" (UID: "188761e4-5b30-40cc-85ec-f4c9b77be511"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:19:58.754875 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:58.754795 2546 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42b48ad0-8671-4297-8af7-3b0cb3e6b87e-openshift-service-ca\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:19:58.754875 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:58.754855 2546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/188761e4-5b30-40cc-85ec-f4c9b77be511-plugins-config-volume\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:19:58.756316 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:58.756288 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/188761e4-5b30-40cc-85ec-f4c9b77be511-kube-api-access-q9nkw" (OuterVolumeSpecName: "kube-api-access-q9nkw") pod "188761e4-5b30-40cc-85ec-f4c9b77be511" (UID: "188761e4-5b30-40cc-85ec-f4c9b77be511"). InnerVolumeSpecName "kube-api-access-q9nkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:19:58.756601 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:58.756562 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42b48ad0-8671-4297-8af7-3b0cb3e6b87e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "42b48ad0-8671-4297-8af7-3b0cb3e6b87e" (UID: "42b48ad0-8671-4297-8af7-3b0cb3e6b87e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 20:19:58.757163 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:58.757135 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42b48ad0-8671-4297-8af7-3b0cb3e6b87e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "42b48ad0-8671-4297-8af7-3b0cb3e6b87e" (UID: "42b48ad0-8671-4297-8af7-3b0cb3e6b87e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 20:19:58.757685 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:58.757667 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42b48ad0-8671-4297-8af7-3b0cb3e6b87e-kube-api-access-ghx5x" (OuterVolumeSpecName: "kube-api-access-ghx5x") pod "42b48ad0-8671-4297-8af7-3b0cb3e6b87e" (UID: "42b48ad0-8671-4297-8af7-3b0cb3e6b87e"). InnerVolumeSpecName "kube-api-access-ghx5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:19:58.856315 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:58.856268 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9nkw\" (UniqueName: \"kubernetes.io/projected/188761e4-5b30-40cc-85ec-f4c9b77be511-kube-api-access-q9nkw\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:19:58.856315 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:58.856302 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghx5x\" (UniqueName: \"kubernetes.io/projected/42b48ad0-8671-4297-8af7-3b0cb3e6b87e-kube-api-access-ghx5x\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:19:58.856315 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:58.856317 2546 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42b48ad0-8671-4297-8af7-3b0cb3e6b87e-metrics-certs\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:19:58.856671 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:58.856329 2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42b48ad0-8671-4297-8af7-3b0cb3e6b87e-maas-api-tls\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:19:58.942239 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:58.942166 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:19:59.041890 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:59.041866 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:19:59.599206 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:59.599175 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3e1dd-545f764567-7qgrd" Sep 11 20:19:59.599206 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:59.599189 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3e1dd-545f764567-7qgrd" event={"ID":"42b48ad0-8671-4297-8af7-3b0cb3e6b87e","Type":"ContainerDied","Data":"ab699a15f1973ce0b89ba6c9fdad22b8d36eb336b828034e1ad5ede9d2d696f3"} Sep 11 20:19:59.599696 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:59.599254 2546 scope.go:117] "RemoveContainer" containerID="cecd0a66fc9565732da59084cbbeb41055e2b3962582acacaf4a68eec1aad5b9" Sep 11 20:19:59.600700 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:59.600671 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b3e1dd-bff5699cd-dv665" event={"ID":"188761e4-5b30-40cc-85ec-f4c9b77be511","Type":"ContainerDied","Data":"faf67a33158fb654c77e963f93b99d74298ec0ef0f705bde5e1ac42e35b5592c"} Sep 11 20:19:59.600793 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:59.600748 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b3e1dd-bff5699cd-dv665" Sep 11 20:19:59.609211 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:59.609188 2546 scope.go:117] "RemoveContainer" containerID="8729ea5d0fc95ba31d3b171ac9dd28b6da203c8d16c57e792ea35c613559e32e" Sep 11 20:19:59.625872 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:59.625849 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b3e1dd-545f764567-7qgrd"] Sep 11 20:19:59.631853 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:59.631833 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b3e1dd-545f764567-7qgrd"] Sep 11 20:19:59.641859 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:59.641833 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b3e1dd-bff5699cd-dv665"] Sep 11 20:19:59.644926 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:59.644903 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b3e1dd-bff5699cd-dv665"] Sep 11 20:19:59.918423 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:19:59.918322 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:20:00.457148 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:00.457105 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="188761e4-5b30-40cc-85ec-f4c9b77be511" path="/var/lib/kubelet/pods/188761e4-5b30-40cc-85ec-f4c9b77be511/volumes" Sep 11 20:20:00.457597 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:00.457561 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42b48ad0-8671-4297-8af7-3b0cb3e6b87e" path="/var/lib/kubelet/pods/42b48ad0-8671-4297-8af7-3b0cb3e6b87e/volumes" Sep 11 20:20:00.620738 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:00.620686 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:20:01.382081 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:01.382053 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:20:02.112400 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:02.112365 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:20:02.812004 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:02.811962 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:20:03.723315 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:03.723281 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:20:04.014336 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:04.014255 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:20:04.412925 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:04.412897 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:20:26.046185 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:26.046154 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs"] Sep 11 20:20:26.049024 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:26.046417 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs" podUID="1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5" containerName="istio-proxy" containerID="cri-o://9055e7b7f97b3c323af75d55c946da400516a5e0a2950196cf92d9c0b81b2b86" gracePeriod=30 Sep 11 20:20:26.066916 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:26.066872 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:20:27.228710 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:27.228679 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:20:31.309856 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:31.309791 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs" Sep 11 20:20:31.397533 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:31.397494 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5-workload-certs\") pod \"1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5\" (UID: \"1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5\") " Sep 11 20:20:31.397726 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:31.397611 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5-istio-podinfo\") pod \"1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5\" (UID: \"1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5\") " Sep 11 20:20:31.397726 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:31.397635 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5-istio-token\") pod \"1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5\" (UID: \"1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5\") " Sep 11 20:20:31.397726 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:31.397689 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5-workload-socket\") pod \"1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5\" (UID: \"1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5\") " Sep 11 20:20:31.397968 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:31.397775 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5-istiod-ca-cert\") pod \"1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5\" (UID: \"1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5\") " Sep 11 20:20:31.397968 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:31.397807 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5" (UID: "1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:20:31.397968 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:31.397837 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5-istio-envoy\") pod \"1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5\" (UID: \"1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5\") " Sep 11 20:20:31.397968 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:31.397888 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f6bfp\" (UniqueName: \"kubernetes.io/projected/1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5-kube-api-access-f6bfp\") pod \"1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5\" (UID: \"1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5\") " Sep 11 20:20:31.397968 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:31.397918 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5-istio-data\") pod \"1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5\" (UID: \"1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5\") " Sep 11 20:20:31.397968 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:31.397942 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5-credential-socket\") pod \"1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5\" (UID: \"1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5\") " Sep 11 20:20:31.398305 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:31.397971 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5" (UID: "1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:20:31.398305 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:31.398153 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5" (UID: "1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:20:31.398305 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:31.398179 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5-istio-data" (OuterVolumeSpecName: "istio-data") pod "1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5" (UID: "1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:20:31.398305 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:31.398273 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5" (UID: "1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:20:31.398535 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:31.398388 2546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5-workload-certs\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:20:31.398535 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:31.398405 2546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5-workload-socket\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:20:31.398535 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:31.398419 2546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5-istiod-ca-cert\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:20:31.398535 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:31.398434 2546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5-istio-data\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:20:31.398535 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:31.398446 2546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5-credential-socket\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:20:31.399949 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:31.399915 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5" (UID: "1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 20:20:31.399949 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:31.399935 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5" (UID: "1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:20:31.400081 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:31.399988 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5-istio-token" (OuterVolumeSpecName: "istio-token") pod "1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5" (UID: "1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:20:31.400163 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:31.400143 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5-kube-api-access-f6bfp" (OuterVolumeSpecName: "kube-api-access-f6bfp") pod "1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5" (UID: "1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5"). InnerVolumeSpecName "kube-api-access-f6bfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:20:31.409307 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:31.409281 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:20:31.499248 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:31.499205 2546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5-istio-podinfo\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:20:31.499248 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:31.499237 2546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5-istio-token\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:20:31.499248 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:31.499251 2546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5-istio-envoy\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:20:31.499484 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:31.499263 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6bfp\" (UniqueName: \"kubernetes.io/projected/1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5-kube-api-access-f6bfp\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:20:31.781991 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:31.781958 2546 generic.go:356] "Generic (PLEG): container finished" podID="1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5" containerID="9055e7b7f97b3c323af75d55c946da400516a5e0a2950196cf92d9c0b81b2b86" exitCode=0 Sep 11 20:20:31.782198 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:31.782012 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs" Sep 11 20:20:31.782198 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:31.782048 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs" event={"ID":"1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5","Type":"ContainerDied","Data":"9055e7b7f97b3c323af75d55c946da400516a5e0a2950196cf92d9c0b81b2b86"} Sep 11 20:20:31.782198 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:31.782102 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs" event={"ID":"1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5","Type":"ContainerDied","Data":"de6114e56300d14242b77715d64162721d5c89f26c5560279b89ba455278a3b7"} Sep 11 20:20:31.782198 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:31.782124 2546 scope.go:117] "RemoveContainer" containerID="9055e7b7f97b3c323af75d55c946da400516a5e0a2950196cf92d9c0b81b2b86" Sep 11 20:20:31.792527 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:31.792507 2546 scope.go:117] "RemoveContainer" containerID="9055e7b7f97b3c323af75d55c946da400516a5e0a2950196cf92d9c0b81b2b86" Sep 11 20:20:31.792841 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:20:31.792823 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9055e7b7f97b3c323af75d55c946da400516a5e0a2950196cf92d9c0b81b2b86\": container with ID starting with 9055e7b7f97b3c323af75d55c946da400516a5e0a2950196cf92d9c0b81b2b86 not found: ID does not exist" containerID="9055e7b7f97b3c323af75d55c946da400516a5e0a2950196cf92d9c0b81b2b86" Sep 11 20:20:31.792899 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:31.792849 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9055e7b7f97b3c323af75d55c946da400516a5e0a2950196cf92d9c0b81b2b86"} err="failed to get container status \"9055e7b7f97b3c323af75d55c946da400516a5e0a2950196cf92d9c0b81b2b86\": rpc error: code = NotFound desc = could not find container \"9055e7b7f97b3c323af75d55c946da400516a5e0a2950196cf92d9c0b81b2b86\": container with ID starting with 9055e7b7f97b3c323af75d55c946da400516a5e0a2950196cf92d9c0b81b2b86 not found: ID does not exist" Sep 11 20:20:31.805126 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:31.805100 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs"] Sep 11 20:20:31.808703 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:31.808681 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-279dba-openshift-default-7b8cbfc954-bprgs"] Sep 11 20:20:32.451170 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:32.451139 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5" path="/var/lib/kubelet/pods/1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5/volumes" Sep 11 20:20:36.516005 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:36.515961 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:20:38.722288 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:38.722254 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:20:40.462274 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:40.462243 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:20:41.110679 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:41.110642 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:20:44.973840 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:44.973808 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:20:46.579293 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:46.579086 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:20:46.769897 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:46.769855 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:20:46.870856 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:46.870808 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:20:47.055980 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:47.055951 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:20:47.550746 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:47.550715 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:20:48.342624 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:48.342595 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:20:49.119659 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:49.119630 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:20:49.781305 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:49.781271 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:20:50.448804 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:50.448769 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:20:51.206017 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:51.205982 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:20:51.886116 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:51.886081 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:20:52.535943 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:52.535910 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:20:57.976419 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:57.976384 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:20:58.012316 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:58.012287 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4f9558-6d94fbdbc8-x6znq"] Sep 11 20:20:58.012545 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:58.012515 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4f9558-6d94fbdbc8-x6znq" podUID="19be5c5d-a319-43c1-bc08-976f128fdccc" containerName="payload-pre-processing" containerID="cri-o://5125f3b52c0387281d8c8a2669627035e1934872e6f5b53cf4406588906ba21a" gracePeriod=30 Sep 11 20:20:58.076569 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:58.076539 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:20:58.177312 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:58.177281 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:20:58.274012 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:58.273986 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:20:58.277029 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:58.277005 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4f9558-6d94fbdbc8-x6znq" Sep 11 20:20:58.369989 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:58.369960 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dmsgz\" (UniqueName: \"kubernetes.io/projected/19be5c5d-a319-43c1-bc08-976f128fdccc-kube-api-access-dmsgz\") pod \"19be5c5d-a319-43c1-bc08-976f128fdccc\" (UID: \"19be5c5d-a319-43c1-bc08-976f128fdccc\") " Sep 11 20:20:58.370204 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:58.370093 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19be5c5d-a319-43c1-bc08-976f128fdccc-plugins-config-volume\") pod \"19be5c5d-a319-43c1-bc08-976f128fdccc\" (UID: \"19be5c5d-a319-43c1-bc08-976f128fdccc\") " Sep 11 20:20:58.370466 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:58.370439 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19be5c5d-a319-43c1-bc08-976f128fdccc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "19be5c5d-a319-43c1-bc08-976f128fdccc" (UID: "19be5c5d-a319-43c1-bc08-976f128fdccc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:20:58.372122 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:58.372100 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19be5c5d-a319-43c1-bc08-976f128fdccc-kube-api-access-dmsgz" (OuterVolumeSpecName: "kube-api-access-dmsgz") pod "19be5c5d-a319-43c1-bc08-976f128fdccc" (UID: "19be5c5d-a319-43c1-bc08-976f128fdccc"). InnerVolumeSpecName "kube-api-access-dmsgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:20:58.471817 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:58.471786 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmsgz\" (UniqueName: \"kubernetes.io/projected/19be5c5d-a319-43c1-bc08-976f128fdccc-kube-api-access-dmsgz\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:20:58.471817 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:58.471813 2546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19be5c5d-a319-43c1-bc08-976f128fdccc-plugins-config-volume\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:20:58.530824 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:58.530750 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:20:58.631655 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:58.631626 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:20:58.958267 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:58.958222 2546 generic.go:356] "Generic (PLEG): container finished" podID="19be5c5d-a319-43c1-bc08-976f128fdccc" containerID="5125f3b52c0387281d8c8a2669627035e1934872e6f5b53cf4406588906ba21a" exitCode=0 Sep 11 20:20:58.958467 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:58.958307 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4f9558-6d94fbdbc8-x6znq" Sep 11 20:20:58.958467 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:58.958295 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4f9558-6d94fbdbc8-x6znq" event={"ID":"19be5c5d-a319-43c1-bc08-976f128fdccc","Type":"ContainerDied","Data":"5125f3b52c0387281d8c8a2669627035e1934872e6f5b53cf4406588906ba21a"} Sep 11 20:20:58.958875 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:58.958778 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4f9558-6d94fbdbc8-x6znq" event={"ID":"19be5c5d-a319-43c1-bc08-976f128fdccc","Type":"ContainerDied","Data":"ba6beac6285f8cc457cab10bf7e7cbc8befb618ec8983181dfeb475000eab4fc"} Sep 11 20:20:58.958875 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:58.958833 2546 scope.go:117] "RemoveContainer" containerID="5125f3b52c0387281d8c8a2669627035e1934872e6f5b53cf4406588906ba21a" Sep 11 20:20:58.968901 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:58.968885 2546 scope.go:117] "RemoveContainer" containerID="5125f3b52c0387281d8c8a2669627035e1934872e6f5b53cf4406588906ba21a" Sep 11 20:20:58.969152 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:20:58.969134 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5125f3b52c0387281d8c8a2669627035e1934872e6f5b53cf4406588906ba21a\": container with ID starting with 5125f3b52c0387281d8c8a2669627035e1934872e6f5b53cf4406588906ba21a not found: ID does not exist" containerID="5125f3b52c0387281d8c8a2669627035e1934872e6f5b53cf4406588906ba21a" Sep 11 20:20:58.969202 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:58.969159 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5125f3b52c0387281d8c8a2669627035e1934872e6f5b53cf4406588906ba21a"} err="failed to get container status \"5125f3b52c0387281d8c8a2669627035e1934872e6f5b53cf4406588906ba21a\": rpc error: code = NotFound desc = could not find container \"5125f3b52c0387281d8c8a2669627035e1934872e6f5b53cf4406588906ba21a\": container with ID starting with 5125f3b52c0387281d8c8a2669627035e1934872e6f5b53cf4406588906ba21a not found: ID does not exist" Sep 11 20:20:58.978202 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:58.978178 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4f9558-6d94fbdbc8-x6znq"] Sep 11 20:20:58.980345 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:58.980324 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4f9558-6d94fbdbc8-x6znq"] Sep 11 20:20:59.384270 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:59.384240 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:20:59.487531 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:20:59.487489 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:00.205896 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:00.205851 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:00.451890 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:00.451857 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19be5c5d-a319-43c1-bc08-976f128fdccc" path="/var/lib/kubelet/pods/19be5c5d-a319-43c1-bc08-976f128fdccc/volumes" Sep 11 20:21:00.689783 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:00.689752 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:01.019521 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:01.019442 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:01.708496 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:01.708466 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:02.514869 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:02.514835 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:03.282195 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:03.282165 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:04.338344 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:04.338314 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:04.402032 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:04.402005 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:04.603795 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:04.603709 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:05.093191 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:05.093155 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:05.439778 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:05.439691 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s"] Sep 11 20:21:05.440416 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:05.440394 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5" containerName="istio-proxy" Sep 11 20:21:05.440416 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:05.440418 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5" containerName="istio-proxy" Sep 11 20:21:05.440605 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:05.440435 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42b48ad0-8671-4297-8af7-3b0cb3e6b87e" containerName="maas-api" Sep 11 20:21:05.440605 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:05.440444 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="42b48ad0-8671-4297-8af7-3b0cb3e6b87e" containerName="maas-api" Sep 11 20:21:05.440605 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:05.440468 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19be5c5d-a319-43c1-bc08-976f128fdccc" containerName="payload-pre-processing" Sep 11 20:21:05.440605 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:05.440477 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="19be5c5d-a319-43c1-bc08-976f128fdccc" containerName="payload-pre-processing" Sep 11 20:21:05.440605 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:05.440497 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="188761e4-5b30-40cc-85ec-f4c9b77be511" containerName="payload-pre-processing" Sep 11 20:21:05.440605 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:05.440505 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="188761e4-5b30-40cc-85ec-f4c9b77be511" containerName="payload-pre-processing" Sep 11 20:21:05.440916 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:05.440624 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="19be5c5d-a319-43c1-bc08-976f128fdccc" containerName="payload-pre-processing" Sep 11 20:21:05.440916 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:05.440639 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="42b48ad0-8671-4297-8af7-3b0cb3e6b87e" containerName="maas-api" Sep 11 20:21:05.440916 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:05.440650 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="188761e4-5b30-40cc-85ec-f4c9b77be511" containerName="payload-pre-processing" Sep 11 20:21:05.440916 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:05.440666 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c1e9a89-57d7-4bf2-aa27-ebbea5f86dd5" containerName="istio-proxy" Sep 11 20:21:05.444155 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:05.444134 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s" Sep 11 20:21:05.447099 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:05.447075 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-fcb0e4-openshift-default-dockercfg-jrpvw\"" Sep 11 20:21:05.456086 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:05.456063 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s"] Sep 11 20:21:05.472472 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:05.472448 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:05.545856 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:05.545821 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b559f06b-d917-4fb4-a548-31e824e58070-istio-podinfo\") pod \"e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s\" (UID: \"b559f06b-d917-4fb4-a548-31e824e58070\") " pod="openshift-ingress/e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s" Sep 11 20:21:05.545856 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:05.545860 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nfkm\" (UniqueName: \"kubernetes.io/projected/b559f06b-d917-4fb4-a548-31e824e58070-kube-api-access-2nfkm\") pod \"e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s\" (UID: \"b559f06b-d917-4fb4-a548-31e824e58070\") " pod="openshift-ingress/e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s" Sep 11 20:21:05.546111 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:05.545914 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b559f06b-d917-4fb4-a548-31e824e58070-workload-socket\") pod \"e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s\" (UID: \"b559f06b-d917-4fb4-a548-31e824e58070\") " pod="openshift-ingress/e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s" Sep 11 20:21:05.546111 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:05.545953 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b559f06b-d917-4fb4-a548-31e824e58070-istiod-ca-cert\") pod \"e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s\" (UID: \"b559f06b-d917-4fb4-a548-31e824e58070\") " pod="openshift-ingress/e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s" Sep 11 20:21:05.546111 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:05.546092 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b559f06b-d917-4fb4-a548-31e824e58070-istio-envoy\") pod \"e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s\" (UID: \"b559f06b-d917-4fb4-a548-31e824e58070\") " pod="openshift-ingress/e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s" Sep 11 20:21:05.546295 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:05.546164 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b559f06b-d917-4fb4-a548-31e824e58070-istio-data\") pod \"e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s\" (UID: \"b559f06b-d917-4fb4-a548-31e824e58070\") " pod="openshift-ingress/e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s" Sep 11 20:21:05.546295 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:05.546213 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b559f06b-d917-4fb4-a548-31e824e58070-credential-socket\") pod \"e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s\" (UID: \"b559f06b-d917-4fb4-a548-31e824e58070\") " pod="openshift-ingress/e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s" Sep 11 20:21:05.546295 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:05.546272 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b559f06b-d917-4fb4-a548-31e824e58070-istio-token\") pod \"e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s\" (UID: \"b559f06b-d917-4fb4-a548-31e824e58070\") " pod="openshift-ingress/e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s" Sep 11 20:21:05.546453 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:05.546302 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b559f06b-d917-4fb4-a548-31e824e58070-workload-certs\") pod \"e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s\" (UID: \"b559f06b-d917-4fb4-a548-31e824e58070\") " pod="openshift-ingress/e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s" Sep 11 20:21:05.647865 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:05.647831 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b559f06b-d917-4fb4-a548-31e824e58070-istiod-ca-cert\") pod \"e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s\" (UID: \"b559f06b-d917-4fb4-a548-31e824e58070\") " pod="openshift-ingress/e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s" Sep 11 20:21:05.648026 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:05.647902 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b559f06b-d917-4fb4-a548-31e824e58070-istio-envoy\") pod \"e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s\" (UID: \"b559f06b-d917-4fb4-a548-31e824e58070\") " pod="openshift-ingress/e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s" Sep 11 20:21:05.648026 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:05.647951 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b559f06b-d917-4fb4-a548-31e824e58070-istio-data\") pod \"e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s\" (UID: \"b559f06b-d917-4fb4-a548-31e824e58070\") " pod="openshift-ingress/e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s" Sep 11 20:21:05.648026 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:05.647984 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b559f06b-d917-4fb4-a548-31e824e58070-credential-socket\") pod \"e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s\" (UID: \"b559f06b-d917-4fb4-a548-31e824e58070\") " pod="openshift-ingress/e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s" Sep 11 20:21:05.648135 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:05.648024 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b559f06b-d917-4fb4-a548-31e824e58070-istio-token\") pod \"e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s\" (UID: \"b559f06b-d917-4fb4-a548-31e824e58070\") " pod="openshift-ingress/e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s" Sep 11 20:21:05.648135 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:05.648055 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b559f06b-d917-4fb4-a548-31e824e58070-workload-certs\") pod \"e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s\" (UID: \"b559f06b-d917-4fb4-a548-31e824e58070\") " pod="openshift-ingress/e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s" Sep 11 20:21:05.648135 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:05.648095 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b559f06b-d917-4fb4-a548-31e824e58070-istio-podinfo\") pod \"e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s\" (UID: \"b559f06b-d917-4fb4-a548-31e824e58070\") " pod="openshift-ingress/e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s" Sep 11 20:21:05.648135 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:05.648126 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nfkm\" (UniqueName: \"kubernetes.io/projected/b559f06b-d917-4fb4-a548-31e824e58070-kube-api-access-2nfkm\") pod \"e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s\" (UID: \"b559f06b-d917-4fb4-a548-31e824e58070\") " pod="openshift-ingress/e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s" Sep 11 20:21:05.648261 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:05.648173 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b559f06b-d917-4fb4-a548-31e824e58070-workload-socket\") pod \"e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s\" (UID: \"b559f06b-d917-4fb4-a548-31e824e58070\") " pod="openshift-ingress/e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s" Sep 11 20:21:05.648485 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:05.648451 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b559f06b-d917-4fb4-a548-31e824e58070-workload-socket\") pod \"e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s\" (UID: \"b559f06b-d917-4fb4-a548-31e824e58070\") " pod="openshift-ingress/e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s" Sep 11 20:21:05.648665 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:05.648644 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b559f06b-d917-4fb4-a548-31e824e58070-istiod-ca-cert\") pod \"e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s\" (UID: \"b559f06b-d917-4fb4-a548-31e824e58070\") " pod="openshift-ingress/e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s" Sep 11 20:21:05.648747 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:05.648726 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b559f06b-d917-4fb4-a548-31e824e58070-workload-certs\") pod \"e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s\" (UID: \"b559f06b-d917-4fb4-a548-31e824e58070\") " pod="openshift-ingress/e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s" Sep 11 20:21:05.648852 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:05.648825 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b559f06b-d917-4fb4-a548-31e824e58070-istio-data\") pod \"e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s\" (UID: \"b559f06b-d917-4fb4-a548-31e824e58070\") " pod="openshift-ingress/e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s" Sep 11 20:21:05.648918 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:05.648874 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b559f06b-d917-4fb4-a548-31e824e58070-credential-socket\") pod \"e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s\" (UID: \"b559f06b-d917-4fb4-a548-31e824e58070\") " pod="openshift-ingress/e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s" Sep 11 20:21:05.650398 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:05.650375 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b559f06b-d917-4fb4-a548-31e824e58070-istio-envoy\") pod \"e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s\" (UID: \"b559f06b-d917-4fb4-a548-31e824e58070\") " pod="openshift-ingress/e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s" Sep 11 20:21:05.650923 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:05.650904 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b559f06b-d917-4fb4-a548-31e824e58070-istio-podinfo\") pod \"e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s\" (UID: \"b559f06b-d917-4fb4-a548-31e824e58070\") " pod="openshift-ingress/e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s" Sep 11 20:21:05.655784 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:05.655760 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b559f06b-d917-4fb4-a548-31e824e58070-istio-token\") pod \"e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s\" (UID: \"b559f06b-d917-4fb4-a548-31e824e58070\") " pod="openshift-ingress/e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s" Sep 11 20:21:05.656021 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:05.656005 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nfkm\" (UniqueName: \"kubernetes.io/projected/b559f06b-d917-4fb4-a548-31e824e58070-kube-api-access-2nfkm\") pod \"e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s\" (UID: \"b559f06b-d917-4fb4-a548-31e824e58070\") " pod="openshift-ingress/e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s" Sep 11 20:21:05.755481 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:05.755412 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s" Sep 11 20:21:05.791420 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:05.791392 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:06.029985 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:06.029956 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s"] Sep 11 20:21:06.031727 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:21:06.031695 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb559f06b_d917_4fb4_a548_31e824e58070.slice/crio-21cf8f2cc94d2840fe88a689cd01f16c49fcc03893b1a063dd89f06ceb49ccb2 WatchSource:0}: Error finding container 21cf8f2cc94d2840fe88a689cd01f16c49fcc03893b1a063dd89f06ceb49ccb2: Status 404 returned error can't find the container with id 21cf8f2cc94d2840fe88a689cd01f16c49fcc03893b1a063dd89f06ceb49ccb2 Sep 11 20:21:06.034034 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:06.034002 2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 20:21:06.034120 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:06.034068 2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 20:21:06.034120 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:06.034096 2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 20:21:07.006015 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:07.005982 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s" event={"ID":"b559f06b-d917-4fb4-a548-31e824e58070","Type":"ContainerStarted","Data":"48b40b50bb4ec3d4dd84e5f88be94d0f05deb69df33942df7c6b8cbd95b7de93"} Sep 11 20:21:07.006398 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:07.006030 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s" event={"ID":"b559f06b-d917-4fb4-a548-31e824e58070","Type":"ContainerStarted","Data":"21cf8f2cc94d2840fe88a689cd01f16c49fcc03893b1a063dd89f06ceb49ccb2"} Sep 11 20:21:07.024952 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:07.024856 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s" podStartSLOduration=2.024839076 podStartE2EDuration="2.024839076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:21:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:21:07.023998893 +0000 UTC m=+1237.072049237" watchObservedRunningTime="2026-09-11 20:21:07.024839076 +0000 UTC m=+1237.072889423" Sep 11 20:21:07.755937 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:07.755900 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s" Sep 11 20:21:07.761540 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:07.761514 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s" Sep 11 20:21:08.012043 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:08.011952 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s" Sep 11 20:21:08.012981 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:08.012957 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s" Sep 11 20:21:08.232540 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:08.232506 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:08.663006 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:08.662922 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:08.943602 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:08.943486 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:09.040927 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:09.040896 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:09.156569 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:09.156517 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:09.921295 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:09.921265 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:10.702872 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:10.702820 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:11.477430 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:11.477396 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:12.283646 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:12.283611 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:12.960027 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:12.959993 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:13.022125 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:13.022093 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:13.210802 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:13.210711 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:13.543093 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:13.543016 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:13.645877 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:13.645848 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:13.657625 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:13.657593 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fcb0e4-755659c658-t9zvt"] Sep 11 20:21:13.661788 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:13.661769 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fcb0e4-755659c658-t9zvt" Sep 11 20:21:13.663906 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:13.663889 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-fcb0e4\"" Sep 11 20:21:13.663972 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:13.663923 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-fcb0e4-dockercfg-xxpfk\"" Sep 11 20:21:13.673952 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:13.673932 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fcb0e4-755659c658-t9zvt"] Sep 11 20:21:13.692336 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:13.692307 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-fcb0e4-5497ffdf9-9zjdw"] Sep 11 20:21:13.696837 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:13.696814 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fcb0e4-5497ffdf9-9zjdw" Sep 11 20:21:13.703623 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:13.703601 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-fcb0e4-5497ffdf9-9zjdw"] Sep 11 20:21:13.727399 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:13.727369 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm9bt\" (UniqueName: \"kubernetes.io/projected/e34b67c7-d6e4-4819-a353-76004387dc4b-kube-api-access-fm9bt\") pod \"payload-pre-processing-e2e-shared-a-w3-fcb0e4-755659c658-t9zvt\" (UID: \"e34b67c7-d6e4-4819-a353-76004387dc4b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fcb0e4-755659c658-t9zvt" Sep 11 20:21:13.727506 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:13.727455 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/461ac6f0-a3e8-4526-849a-290a9872e785-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-fcb0e4-5497ffdf9-9zjdw\" (UID: \"461ac6f0-a3e8-4526-849a-290a9872e785\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fcb0e4-5497ffdf9-9zjdw" Sep 11 20:21:13.727570 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:13.727517 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e34b67c7-d6e4-4819-a353-76004387dc4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-fcb0e4-755659c658-t9zvt\" (UID: \"e34b67c7-d6e4-4819-a353-76004387dc4b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fcb0e4-755659c658-t9zvt" Sep 11 20:21:13.727633 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:13.727570 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvk9b\" (UniqueName: \"kubernetes.io/projected/461ac6f0-a3e8-4526-849a-290a9872e785-kube-api-access-lvk9b\") pod \"payload-processing-e2e-shared-a-w3-fcb0e4-5497ffdf9-9zjdw\" (UID: \"461ac6f0-a3e8-4526-849a-290a9872e785\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fcb0e4-5497ffdf9-9zjdw" Sep 11 20:21:13.828376 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:13.828248 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm9bt\" (UniqueName: \"kubernetes.io/projected/e34b67c7-d6e4-4819-a353-76004387dc4b-kube-api-access-fm9bt\") pod \"payload-pre-processing-e2e-shared-a-w3-fcb0e4-755659c658-t9zvt\" (UID: \"e34b67c7-d6e4-4819-a353-76004387dc4b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fcb0e4-755659c658-t9zvt" Sep 11 20:21:13.828376 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:13.828333 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/461ac6f0-a3e8-4526-849a-290a9872e785-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-fcb0e4-5497ffdf9-9zjdw\" (UID: \"461ac6f0-a3e8-4526-849a-290a9872e785\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fcb0e4-5497ffdf9-9zjdw" Sep 11 20:21:13.828376 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:13.828369 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e34b67c7-d6e4-4819-a353-76004387dc4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-fcb0e4-755659c658-t9zvt\" (UID: \"e34b67c7-d6e4-4819-a353-76004387dc4b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fcb0e4-755659c658-t9zvt" Sep 11 20:21:13.828695 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:13.828403 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvk9b\" (UniqueName: \"kubernetes.io/projected/461ac6f0-a3e8-4526-849a-290a9872e785-kube-api-access-lvk9b\") pod \"payload-processing-e2e-shared-a-w3-fcb0e4-5497ffdf9-9zjdw\" (UID: \"461ac6f0-a3e8-4526-849a-290a9872e785\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fcb0e4-5497ffdf9-9zjdw" Sep 11 20:21:13.829477 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:13.829448 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/461ac6f0-a3e8-4526-849a-290a9872e785-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-fcb0e4-5497ffdf9-9zjdw\" (UID: \"461ac6f0-a3e8-4526-849a-290a9872e785\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fcb0e4-5497ffdf9-9zjdw" Sep 11 20:21:13.829927 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:13.829905 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e34b67c7-d6e4-4819-a353-76004387dc4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-fcb0e4-755659c658-t9zvt\" (UID: \"e34b67c7-d6e4-4819-a353-76004387dc4b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fcb0e4-755659c658-t9zvt" Sep 11 20:21:13.835910 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:13.835881 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm9bt\" (UniqueName: \"kubernetes.io/projected/e34b67c7-d6e4-4819-a353-76004387dc4b-kube-api-access-fm9bt\") pod \"payload-pre-processing-e2e-shared-a-w3-fcb0e4-755659c658-t9zvt\" (UID: \"e34b67c7-d6e4-4819-a353-76004387dc4b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fcb0e4-755659c658-t9zvt" Sep 11 20:21:13.836050 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:13.836007 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvk9b\" (UniqueName: \"kubernetes.io/projected/461ac6f0-a3e8-4526-849a-290a9872e785-kube-api-access-lvk9b\") pod \"payload-processing-e2e-shared-a-w3-fcb0e4-5497ffdf9-9zjdw\" (UID: \"461ac6f0-a3e8-4526-849a-290a9872e785\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fcb0e4-5497ffdf9-9zjdw" Sep 11 20:21:13.973977 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:13.973939 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fcb0e4-755659c658-t9zvt" Sep 11 20:21:14.010124 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:14.010091 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fcb0e4-5497ffdf9-9zjdw" Sep 11 20:21:14.260177 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:14.260149 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fcb0e4-755659c658-t9zvt"] Sep 11 20:21:14.261514 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:21:14.261488 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode34b67c7_d6e4_4819_a353_76004387dc4b.slice/crio-7900c6390960ee14bc20c1a8bdb8a4d02cb0aca474a410408062f44d0f7f8523 WatchSource:0}: Error finding container 7900c6390960ee14bc20c1a8bdb8a4d02cb0aca474a410408062f44d0f7f8523: Status 404 returned error can't find the container with id 7900c6390960ee14bc20c1a8bdb8a4d02cb0aca474a410408062f44d0f7f8523 Sep 11 20:21:14.313798 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:14.313775 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-fcb0e4-5497ffdf9-9zjdw"] Sep 11 20:21:14.322742 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:21:14.322718 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod461ac6f0_a3e8_4526_849a_290a9872e785.slice/crio-b95f391f0b9d67f9136fe8acf7be5fdca2f02fd9628a0d930cc2cc5856e349f3 WatchSource:0}: Error finding container b95f391f0b9d67f9136fe8acf7be5fdca2f02fd9628a0d930cc2cc5856e349f3: Status 404 returned error can't find the container with id b95f391f0b9d67f9136fe8acf7be5fdca2f02fd9628a0d930cc2cc5856e349f3 Sep 11 20:21:15.070883 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:15.070840 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fcb0e4-5497ffdf9-9zjdw" event={"ID":"461ac6f0-a3e8-4526-849a-290a9872e785","Type":"ContainerStarted","Data":"37bb93cb777e95a16f4d6cbabf08a048099f5df5ce2f7e391525bb25193974e7"} Sep 11 20:21:15.071338 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:15.070910 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fcb0e4-5497ffdf9-9zjdw" event={"ID":"461ac6f0-a3e8-4526-849a-290a9872e785","Type":"ContainerStarted","Data":"b95f391f0b9d67f9136fe8acf7be5fdca2f02fd9628a0d930cc2cc5856e349f3"} Sep 11 20:21:15.071338 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:15.070957 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fcb0e4-5497ffdf9-9zjdw" Sep 11 20:21:15.072564 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:15.072541 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fcb0e4-755659c658-t9zvt" event={"ID":"e34b67c7-d6e4-4819-a353-76004387dc4b","Type":"ContainerStarted","Data":"8c23f22f991afcd2af236a4cef804f0cf6f16501e4d6df2e678ad51f41ec95ab"} Sep 11 20:21:15.072675 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:15.072597 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fcb0e4-755659c658-t9zvt" event={"ID":"e34b67c7-d6e4-4819-a353-76004387dc4b","Type":"ContainerStarted","Data":"7900c6390960ee14bc20c1a8bdb8a4d02cb0aca474a410408062f44d0f7f8523"} Sep 11 20:21:15.072675 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:15.072624 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fcb0e4-755659c658-t9zvt" Sep 11 20:21:15.087369 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:15.087327 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fcb0e4-5497ffdf9-9zjdw" podStartSLOduration=2.087313771 podStartE2EDuration="2.087313771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:21:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:21:15.085896505 +0000 UTC m=+1245.133946850" watchObservedRunningTime="2026-09-11 20:21:15.087313771 +0000 UTC m=+1245.135364114" Sep 11 20:21:15.102835 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:15.102783 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fcb0e4-755659c658-t9zvt" podStartSLOduration=2.10276729 podStartE2EDuration="2.10276729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:21:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:21:15.099986808 +0000 UTC m=+1245.148037150" watchObservedRunningTime="2026-09-11 20:21:15.10276729 +0000 UTC m=+1245.150817632" Sep 11 20:21:26.082290 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:26.082210 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fcb0e4-755659c658-t9zvt" Sep 11 20:21:26.082774 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:26.082370 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fcb0e4-5497ffdf9-9zjdw" Sep 11 20:21:28.289981 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:28.289939 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:29.014917 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:29.014869 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:29.524171 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:29.524141 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:30.628722 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:30.628686 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:31.369007 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:31.368973 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:32.143297 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:32.143255 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:33.146144 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:33.146086 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:33.790308 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:33.790274 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:33.911090 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:33.911051 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:34.111132 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:34.111104 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:34.212142 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:34.212107 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:34.391673 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:34.391562 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:34.673439 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:34.673366 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:34.804081 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:34.804047 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:35.664197 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:35.664166 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:36.444263 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:36.444221 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:37.413068 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:37.413030 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:37.674993 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:37.674913 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:37.769847 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:37.769823 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:38.013604 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:38.013471 2546 patch_prober.go:28] interesting pod/e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Sep 11 20:21:38.013604 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:38.013530 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s" podUID="b559f06b-d917-4fb4-a548-31e824e58070" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Sep 11 20:21:38.219746 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:38.219717 2546 generic.go:356] "Generic (PLEG): container finished" podID="b559f06b-d917-4fb4-a548-31e824e58070" containerID="48b40b50bb4ec3d4dd84e5f88be94d0f05deb69df33942df7c6b8cbd95b7de93" exitCode=0 Sep 11 20:21:38.219932 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:38.219797 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s" event={"ID":"b559f06b-d917-4fb4-a548-31e824e58070","Type":"ContainerDied","Data":"48b40b50bb4ec3d4dd84e5f88be94d0f05deb69df33942df7c6b8cbd95b7de93"} Sep 11 20:21:38.220260 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:38.220243 2546 scope.go:117] "RemoveContainer" containerID="48b40b50bb4ec3d4dd84e5f88be94d0f05deb69df33942df7c6b8cbd95b7de93" Sep 11 20:21:38.221398 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:38.221370 2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 20:21:38.221468 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:38.221455 2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 20:21:38.221507 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:38.221488 2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 20:21:38.230040 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:38.230023 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:39.229822 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:39.229786 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s" event={"ID":"b559f06b-d917-4fb4-a548-31e824e58070","Type":"ContainerStarted","Data":"beb00cb9814cda29b18dd566fd38f5b148027cf79b3b6c1e64c1f2fecad5f977"} Sep 11 20:21:39.444695 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:39.444657 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:39.756246 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:39.756204 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s" Sep 11 20:21:39.756454 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:39.756431 2546 patch_prober.go:28] interesting pod/e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Sep 11 20:21:39.756513 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:39.756494 2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s" podUID="b559f06b-d917-4fb4-a548-31e824e58070" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Sep 11 20:21:40.224050 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:40.224021 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:40.286482 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:40.286435 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:40.405785 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:40.405749 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:40.746601 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:40.745554 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:40.756174 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:40.756141 2546 patch_prober.go:28] interesting pod/e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Sep 11 20:21:40.756427 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:40.756395 2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s" podUID="b559f06b-d917-4fb4-a548-31e824e58070" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Sep 11 20:21:40.840600 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:40.839901 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:41.219219 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:41.219187 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:41.756908 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:41.756872 2546 patch_prober.go:28] interesting pod/e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Sep 11 20:21:41.757346 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:41.756945 2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s" podUID="b559f06b-d917-4fb4-a548-31e824e58070" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Sep 11 20:21:42.346333 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:42.346293 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:42.756436 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:42.756353 2546 patch_prober.go:28] interesting pod/e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Sep 11 20:21:42.756436 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:42.756406 2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s" podUID="b559f06b-d917-4fb4-a548-31e824e58070" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Sep 11 20:21:42.927878 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:42.927840 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:43.119315 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:43.119285 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:43.756450 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:43.756412 2546 patch_prober.go:28] interesting pod/e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Sep 11 20:21:43.756642 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:43.756474 2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s" podUID="b559f06b-d917-4fb4-a548-31e824e58070" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Sep 11 20:21:43.898467 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:43.898431 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:44.760059 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:44.760023 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s" Sep 11 20:21:44.760468 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:44.760444 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s" Sep 11 20:21:44.761005 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:44.760987 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s" Sep 11 20:21:46.857595 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:46.857535 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-9d9abb-564996c49d-2x6rg"] Sep 11 20:21:46.858085 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:46.857796 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9d9abb-564996c49d-2x6rg" podUID="26126c9a-d580-4890-909b-ad74419c775d" containerName="maas-api" containerID="cri-o://d4439ce5346d8b22ee60cb98a9ff790cf68fef4d1db536a779b291a09c26992e" gracePeriod=30 Sep 11 20:21:46.882170 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:46.882147 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:46.960562 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:46.960461 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:47.020191 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:47.020164 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:47.115772 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:47.115691 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:47.118113 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:47.118095 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9d9abb-564996c49d-2x6rg" Sep 11 20:21:47.184189 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:47.184161 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:47.216513 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:47.216482 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6snvp\" (UniqueName: \"kubernetes.io/projected/26126c9a-d580-4890-909b-ad74419c775d-kube-api-access-6snvp\") pod \"26126c9a-d580-4890-909b-ad74419c775d\" (UID: \"26126c9a-d580-4890-909b-ad74419c775d\") " Sep 11 20:21:47.216673 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:47.216544 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26126c9a-d580-4890-909b-ad74419c775d-openshift-service-ca\") pod \"26126c9a-d580-4890-909b-ad74419c775d\" (UID: \"26126c9a-d580-4890-909b-ad74419c775d\") " Sep 11 20:21:47.216673 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:47.216610 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26126c9a-d580-4890-909b-ad74419c775d-maas-api-tls\") pod \"26126c9a-d580-4890-909b-ad74419c775d\" (UID: \"26126c9a-d580-4890-909b-ad74419c775d\") " Sep 11 20:21:47.216673 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:47.216662 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26126c9a-d580-4890-909b-ad74419c775d-metrics-certs\") pod \"26126c9a-d580-4890-909b-ad74419c775d\" (UID: \"26126c9a-d580-4890-909b-ad74419c775d\") " Sep 11 20:21:47.216975 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:47.216951 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26126c9a-d580-4890-909b-ad74419c775d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "26126c9a-d580-4890-909b-ad74419c775d" (UID: "26126c9a-d580-4890-909b-ad74419c775d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:21:47.218699 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:47.218669 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26126c9a-d580-4890-909b-ad74419c775d-kube-api-access-6snvp" (OuterVolumeSpecName: "kube-api-access-6snvp") pod "26126c9a-d580-4890-909b-ad74419c775d" (UID: "26126c9a-d580-4890-909b-ad74419c775d"). InnerVolumeSpecName "kube-api-access-6snvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:21:47.218928 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:47.218897 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26126c9a-d580-4890-909b-ad74419c775d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "26126c9a-d580-4890-909b-ad74419c775d" (UID: "26126c9a-d580-4890-909b-ad74419c775d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 20:21:47.219171 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:47.219152 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26126c9a-d580-4890-909b-ad74419c775d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "26126c9a-d580-4890-909b-ad74419c775d" (UID: "26126c9a-d580-4890-909b-ad74419c775d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 20:21:47.297460 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:47.297423 2546 generic.go:356] "Generic (PLEG): container finished" podID="26126c9a-d580-4890-909b-ad74419c775d" containerID="d4439ce5346d8b22ee60cb98a9ff790cf68fef4d1db536a779b291a09c26992e" exitCode=0 Sep 11 20:21:47.297646 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:47.297467 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9d9abb-564996c49d-2x6rg" Sep 11 20:21:47.297646 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:47.297502 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9d9abb-564996c49d-2x6rg" event={"ID":"26126c9a-d580-4890-909b-ad74419c775d","Type":"ContainerDied","Data":"d4439ce5346d8b22ee60cb98a9ff790cf68fef4d1db536a779b291a09c26992e"} Sep 11 20:21:47.297646 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:47.297555 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9d9abb-564996c49d-2x6rg" event={"ID":"26126c9a-d580-4890-909b-ad74419c775d","Type":"ContainerDied","Data":"94f8bdd478481cd3bdcdd69522314c7339691be70c43bb28d6332031b31a5d70"} Sep 11 20:21:47.297646 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:47.297602 2546 scope.go:117] "RemoveContainer" containerID="d4439ce5346d8b22ee60cb98a9ff790cf68fef4d1db536a779b291a09c26992e" Sep 11 20:21:47.308331 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:47.308313 2546 scope.go:117] "RemoveContainer" containerID="d4439ce5346d8b22ee60cb98a9ff790cf68fef4d1db536a779b291a09c26992e" Sep 11 20:21:47.308634 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:21:47.308611 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4439ce5346d8b22ee60cb98a9ff790cf68fef4d1db536a779b291a09c26992e\": container with ID starting with d4439ce5346d8b22ee60cb98a9ff790cf68fef4d1db536a779b291a09c26992e not found: ID does not exist" containerID="d4439ce5346d8b22ee60cb98a9ff790cf68fef4d1db536a779b291a09c26992e" Sep 11 20:21:47.308712 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:47.308641 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4439ce5346d8b22ee60cb98a9ff790cf68fef4d1db536a779b291a09c26992e"} err="failed to get container status \"d4439ce5346d8b22ee60cb98a9ff790cf68fef4d1db536a779b291a09c26992e\": rpc error: code = NotFound desc = could not find container \"d4439ce5346d8b22ee60cb98a9ff790cf68fef4d1db536a779b291a09c26992e\": container with ID starting with d4439ce5346d8b22ee60cb98a9ff790cf68fef4d1db536a779b291a09c26992e not found: ID does not exist" Sep 11 20:21:47.317599 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:47.317553 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6snvp\" (UniqueName: \"kubernetes.io/projected/26126c9a-d580-4890-909b-ad74419c775d-kube-api-access-6snvp\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:21:47.317683 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:47.317603 2546 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26126c9a-d580-4890-909b-ad74419c775d-openshift-service-ca\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:21:47.317683 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:47.317617 2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26126c9a-d580-4890-909b-ad74419c775d-maas-api-tls\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:21:47.317683 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:47.317628 2546 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26126c9a-d580-4890-909b-ad74419c775d-metrics-certs\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:21:47.319334 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:47.319314 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-9d9abb-564996c49d-2x6rg"] Sep 11 20:21:47.321445 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:47.321426 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-9d9abb-564996c49d-2x6rg"] Sep 11 20:21:47.462047 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:47.461972 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:47.562937 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:47.562904 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:48.237075 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:48.237044 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:48.450818 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:48.450782 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26126c9a-d580-4890-909b-ad74419c775d" path="/var/lib/kubelet/pods/26126c9a-d580-4890-909b-ad74419c775d/volumes" Sep 11 20:21:49.090897 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:49.090866 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:49.800782 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:49.800749 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:50.548052 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:50.548019 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:51.377815 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:51.377712 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:52.171051 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:52.171020 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:52.380229 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:52.380180 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:52.911691 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:52.911658 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:53.893795 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:53.893759 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:53.994436 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:53.994384 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:54.146752 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:54.146601 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs"] Sep 11 20:21:54.146911 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:54.146882 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs" podUID="7947d808-ad70-4b13-b396-4936f2588e44" containerName="istio-proxy" containerID="cri-o://f99c815f2928ef5c20d2d309eb2d3ea160a8fadac65a37cb05028b5523727a8f" gracePeriod=30 Sep 11 20:21:54.174228 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:54.174175 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:54.662672 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:54.662629 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:55.219287 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:55.219253 2546 patch_prober.go:28] interesting pod/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 20:21:55.219672 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:55.219318 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs" podUID="7947d808-ad70-4b13-b396-4936f2588e44" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 20:21:59.371910 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:59.371879 2546 generic.go:356] "Generic (PLEG): container finished" podID="7947d808-ad70-4b13-b396-4936f2588e44" containerID="f99c815f2928ef5c20d2d309eb2d3ea160a8fadac65a37cb05028b5523727a8f" exitCode=0 Sep 11 20:21:59.372348 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:59.371949 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs" event={"ID":"7947d808-ad70-4b13-b396-4936f2588e44","Type":"ContainerDied","Data":"f99c815f2928ef5c20d2d309eb2d3ea160a8fadac65a37cb05028b5523727a8f"} Sep 11 20:21:59.372348 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:59.372011 2546 scope.go:117] "RemoveContainer" containerID="8ae7fc1b8494c0d7e86d8740688a9e8f3391285332a2888fb1be74972b40716d" Sep 11 20:21:59.417353 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:59.417308 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs" Sep 11 20:21:59.426190 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:59.426169 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:59.477659 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:59.477635 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:21:59.545880 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:59.545856 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7947d808-ad70-4b13-b396-4936f2588e44-istio-data\") pod \"7947d808-ad70-4b13-b396-4936f2588e44\" (UID: \"7947d808-ad70-4b13-b396-4936f2588e44\") " Sep 11 20:21:59.545981 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:59.545899 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7947d808-ad70-4b13-b396-4936f2588e44-credential-socket\") pod \"7947d808-ad70-4b13-b396-4936f2588e44\" (UID: \"7947d808-ad70-4b13-b396-4936f2588e44\") " Sep 11 20:21:59.545981 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:59.545922 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7947d808-ad70-4b13-b396-4936f2588e44-workload-certs\") pod \"7947d808-ad70-4b13-b396-4936f2588e44\" (UID: \"7947d808-ad70-4b13-b396-4936f2588e44\") " Sep 11 20:21:59.545981 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:59.545971 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7947d808-ad70-4b13-b396-4936f2588e44-workload-socket\") pod \"7947d808-ad70-4b13-b396-4936f2588e44\" (UID: \"7947d808-ad70-4b13-b396-4936f2588e44\") " Sep 11 20:21:59.546136 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:59.545997 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9wxd\" (UniqueName: \"kubernetes.io/projected/7947d808-ad70-4b13-b396-4936f2588e44-kube-api-access-v9wxd\") pod \"7947d808-ad70-4b13-b396-4936f2588e44\" (UID: \"7947d808-ad70-4b13-b396-4936f2588e44\") " Sep 11 20:21:59.546136 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:59.546056 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7947d808-ad70-4b13-b396-4936f2588e44-istio-podinfo\") pod \"7947d808-ad70-4b13-b396-4936f2588e44\" (UID: \"7947d808-ad70-4b13-b396-4936f2588e44\") " Sep 11 20:21:59.546136 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:59.546079 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7947d808-ad70-4b13-b396-4936f2588e44-istio-envoy\") pod \"7947d808-ad70-4b13-b396-4936f2588e44\" (UID: \"7947d808-ad70-4b13-b396-4936f2588e44\") " Sep 11 20:21:59.546136 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:59.546107 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7947d808-ad70-4b13-b396-4936f2588e44-istiod-ca-cert\") pod \"7947d808-ad70-4b13-b396-4936f2588e44\" (UID: \"7947d808-ad70-4b13-b396-4936f2588e44\") " Sep 11 20:21:59.546339 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:59.546147 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7947d808-ad70-4b13-b396-4936f2588e44-istio-data" (OuterVolumeSpecName: "istio-data") pod "7947d808-ad70-4b13-b396-4936f2588e44" (UID: "7947d808-ad70-4b13-b396-4936f2588e44"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:21:59.546339 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:59.546155 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7947d808-ad70-4b13-b396-4936f2588e44-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7947d808-ad70-4b13-b396-4936f2588e44" (UID: "7947d808-ad70-4b13-b396-4936f2588e44"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:21:59.546339 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:59.546226 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7947d808-ad70-4b13-b396-4936f2588e44-istio-token\") pod \"7947d808-ad70-4b13-b396-4936f2588e44\" (UID: \"7947d808-ad70-4b13-b396-4936f2588e44\") " Sep 11 20:21:59.546339 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:59.546271 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7947d808-ad70-4b13-b396-4936f2588e44-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7947d808-ad70-4b13-b396-4936f2588e44" (UID: "7947d808-ad70-4b13-b396-4936f2588e44"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:21:59.546339 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:59.546286 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7947d808-ad70-4b13-b396-4936f2588e44-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7947d808-ad70-4b13-b396-4936f2588e44" (UID: "7947d808-ad70-4b13-b396-4936f2588e44"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:21:59.546548 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:59.546475 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7947d808-ad70-4b13-b396-4936f2588e44-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7947d808-ad70-4b13-b396-4936f2588e44" (UID: "7947d808-ad70-4b13-b396-4936f2588e44"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:21:59.546671 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:59.546651 2546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7947d808-ad70-4b13-b396-4936f2588e44-workload-socket\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:21:59.546736 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:59.546677 2546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7947d808-ad70-4b13-b396-4936f2588e44-istiod-ca-cert\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:21:59.546736 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:59.546690 2546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7947d808-ad70-4b13-b396-4936f2588e44-istio-data\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:21:59.546736 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:59.546704 2546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7947d808-ad70-4b13-b396-4936f2588e44-credential-socket\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:21:59.546736 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:59.546716 2546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7947d808-ad70-4b13-b396-4936f2588e44-workload-certs\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:21:59.548359 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:59.548329 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7947d808-ad70-4b13-b396-4936f2588e44-istio-token" (OuterVolumeSpecName: "istio-token") pod "7947d808-ad70-4b13-b396-4936f2588e44" (UID: "7947d808-ad70-4b13-b396-4936f2588e44"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:21:59.548359 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:59.548349 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7947d808-ad70-4b13-b396-4936f2588e44-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7947d808-ad70-4b13-b396-4936f2588e44" (UID: "7947d808-ad70-4b13-b396-4936f2588e44"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 20:21:59.548359 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:59.548334 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7947d808-ad70-4b13-b396-4936f2588e44-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7947d808-ad70-4b13-b396-4936f2588e44" (UID: "7947d808-ad70-4b13-b396-4936f2588e44"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:21:59.548522 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:59.548416 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7947d808-ad70-4b13-b396-4936f2588e44-kube-api-access-v9wxd" (OuterVolumeSpecName: "kube-api-access-v9wxd") pod "7947d808-ad70-4b13-b396-4936f2588e44" (UID: "7947d808-ad70-4b13-b396-4936f2588e44"). InnerVolumeSpecName "kube-api-access-v9wxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:21:59.648026 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:59.647997 2546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7947d808-ad70-4b13-b396-4936f2588e44-istio-podinfo\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:21:59.648026 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:59.648024 2546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7947d808-ad70-4b13-b396-4936f2588e44-istio-envoy\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:21:59.648172 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:59.648033 2546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7947d808-ad70-4b13-b396-4936f2588e44-istio-token\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:21:59.648172 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:21:59.648041 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9wxd\" (UniqueName: \"kubernetes.io/projected/7947d808-ad70-4b13-b396-4936f2588e44-kube-api-access-v9wxd\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:22:00.378812 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:22:00.378786 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs" Sep 11 20:22:00.378812 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:22:00.378808 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs" event={"ID":"7947d808-ad70-4b13-b396-4936f2588e44","Type":"ContainerDied","Data":"3d1dfafd92868a9ce50e3e33a9af4d440d414d2119aba89590952ec2a54eba9b"} Sep 11 20:22:00.379284 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:22:00.378861 2546 scope.go:117] "RemoveContainer" containerID="f99c815f2928ef5c20d2d309eb2d3ea160a8fadac65a37cb05028b5523727a8f" Sep 11 20:22:00.404803 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:22:00.404779 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs"] Sep 11 20:22:00.409866 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:22:00.409843 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-9d9abb-openshift-default-77c6d7fb86-7vxgs"] Sep 11 20:22:00.453965 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:22:00.453926 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7947d808-ad70-4b13-b396-4936f2588e44" path="/var/lib/kubelet/pods/7947d808-ad70-4b13-b396-4936f2588e44/volumes" Sep 11 20:22:01.254868 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:22:01.254825 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:22:09.119514 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:22:09.119478 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:22:11.075432 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:22:11.075397 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:22:11.175630 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:22:11.175595 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:22:11.417462 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:22:11.417413 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:22:13.132705 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:22:13.132675 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:22:13.930464 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:22:13.930418 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:22:14.715025 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:22:14.714988 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:22:15.412867 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:22:15.412827 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:22:16.084842 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:22:16.084807 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:22:16.822739 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:22:16.822707 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:22:17.126268 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:22:17.126241 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:22:17.508533 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:22:17.508446 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:22:18.375151 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:22:18.375121 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:22:23.904623 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:22:23.904567 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:22:28.507887 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:22:28.507848 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:22:28.607788 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:22:28.607753 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:22:28.705458 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:22:28.705430 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:22:28.771684 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:22:28.771606 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:22:29.095609 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:22:29.095558 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:22:29.923917 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:22:29.923888 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:22:30.615991 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:22:30.615953 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:22:31.436349 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:22:31.436311 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:22:31.786021 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:22:31.785942 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:22:32.227385 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:22:32.227351 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:22:33.068961 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:22:33.068933 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:22:36.811407 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:22:36.811375 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:23:03.297416 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:23:03.297384 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:23:03.355005 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:23:03.354976 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:23:07.621610 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:23:07.621555 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:23:13.526163 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:23:13.526128 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:23:13.770985 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:23:13.770942 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:23:14.871771 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:23:14.871741 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:23:19.511412 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:23:19.511382 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:23:27.627368 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:23:27.627214 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:23:42.401822 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:23:42.401789 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:23:42.502500 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:23:42.502464 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:23:42.705483 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:23:42.705407 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:23:44.051901 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:23:44.051873 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:23:44.150830 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:23:44.150798 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:23:44.251822 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:23:44.251790 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:23:44.875341 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:23:44.875301 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:23:44.971487 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:23:44.971460 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:23:45.072003 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:23:45.071970 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:23:46.020912 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:23:46.020884 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:23:46.339048 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:23:46.339020 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:23:52.050119 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:23:52.050087 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:23:52.113496 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:23:52.113456 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:23:52.810277 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:23:52.810244 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:23:59.102843 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:23:59.102813 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:23:59.159147 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:23:59.159115 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:23:59.249290 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:23:59.249262 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:23:59.348808 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:23:59.348778 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:24:09.832239 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:24:09.832205 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:24:10.078481 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:24:10.078446 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:24:19.383531 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:24:19.383490 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:24:20.310171 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:24:20.310138 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:24:20.410806 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:24:20.410774 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:24:31.049366 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:24:31.049291 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:24:36.929269 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:24:36.929231 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:24:37.029938 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:24:37.029895 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:24:37.284132 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:24:37.284052 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:24:37.577569 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:24:37.577488 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:24:38.067689 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:24:38.067634 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:24:40.087729 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:24:40.087693 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:24:40.699498 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:24:40.699457 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:24:40.825399 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:24:40.825365 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:24:41.014182 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:24:41.014102 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:24:41.927232 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:24:41.927191 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:24:42.709149 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:24:42.709112 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:24:43.412221 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:24:43.412187 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:24:44.234398 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:24:44.234351 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:24:45.037462 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:24:45.037428 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3faec0-95db6ff75-w9729"] Sep 11 20:24:45.038214 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:24:45.038197 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7947d808-ad70-4b13-b396-4936f2588e44" containerName="istio-proxy" Sep 11 20:24:45.038259 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:24:45.038217 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="7947d808-ad70-4b13-b396-4936f2588e44" containerName="istio-proxy" Sep 11 20:24:45.038259 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:24:45.038238 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7947d808-ad70-4b13-b396-4936f2588e44" containerName="istio-proxy" Sep 11 20:24:45.038259 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:24:45.038247 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="7947d808-ad70-4b13-b396-4936f2588e44" containerName="istio-proxy" Sep 11 20:24:45.038354 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:24:45.038265 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26126c9a-d580-4890-909b-ad74419c775d" containerName="maas-api" Sep 11 20:24:45.038354 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:24:45.038273 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="26126c9a-d580-4890-909b-ad74419c775d" containerName="maas-api" Sep 11 20:24:45.038416 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:24:45.038403 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="26126c9a-d580-4890-909b-ad74419c775d" containerName="maas-api" Sep 11 20:24:45.038462 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:24:45.038415 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="7947d808-ad70-4b13-b396-4936f2588e44" containerName="istio-proxy" Sep 11 20:24:45.038462 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:24:45.038434 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="7947d808-ad70-4b13-b396-4936f2588e44" containerName="istio-proxy" Sep 11 20:24:45.042450 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:24:45.042420 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3faec0-95db6ff75-w9729" Sep 11 20:24:45.044477 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:24:45.044453 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-3faec0-serving-cert\"" Sep 11 20:24:45.051926 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:24:45.051894 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3faec0-95db6ff75-w9729"] Sep 11 20:24:45.073505 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:24:45.073470 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:24:45.155856 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:24:45.155823 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:24:45.209026 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:24:45.208988 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db0b2703-500b-47e2-9165-8808feae0778-metrics-certs\") pod \"maas-api-e2e-api-a-3faec0-95db6ff75-w9729\" (UID: \"db0b2703-500b-47e2-9165-8808feae0778\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3faec0-95db6ff75-w9729" Sep 11 20:24:45.209171 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:24:45.209103 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db0b2703-500b-47e2-9165-8808feae0778-maas-api-tls\") pod \"maas-api-e2e-api-a-3faec0-95db6ff75-w9729\" (UID: \"db0b2703-500b-47e2-9165-8808feae0778\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3faec0-95db6ff75-w9729" Sep 11 20:24:45.209171 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:24:45.209137 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db0b2703-500b-47e2-9165-8808feae0778-openshift-service-ca\") pod \"maas-api-e2e-api-a-3faec0-95db6ff75-w9729\" (UID: \"db0b2703-500b-47e2-9165-8808feae0778\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3faec0-95db6ff75-w9729" Sep 11 20:24:45.209268 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:24:45.209242 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkr5r\" (UniqueName: \"kubernetes.io/projected/db0b2703-500b-47e2-9165-8808feae0778-kube-api-access-fkr5r\") pod \"maas-api-e2e-api-a-3faec0-95db6ff75-w9729\" (UID: \"db0b2703-500b-47e2-9165-8808feae0778\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3faec0-95db6ff75-w9729" Sep 11 20:24:45.310825 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:24:45.310745 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkr5r\" (UniqueName: \"kubernetes.io/projected/db0b2703-500b-47e2-9165-8808feae0778-kube-api-access-fkr5r\") pod \"maas-api-e2e-api-a-3faec0-95db6ff75-w9729\" (UID: \"db0b2703-500b-47e2-9165-8808feae0778\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3faec0-95db6ff75-w9729" Sep 11 20:24:45.310825 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:24:45.310807 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db0b2703-500b-47e2-9165-8808feae0778-metrics-certs\") pod \"maas-api-e2e-api-a-3faec0-95db6ff75-w9729\" (UID: \"db0b2703-500b-47e2-9165-8808feae0778\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3faec0-95db6ff75-w9729" Sep 11 20:24:45.311031 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:24:45.310874 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db0b2703-500b-47e2-9165-8808feae0778-maas-api-tls\") pod \"maas-api-e2e-api-a-3faec0-95db6ff75-w9729\" (UID: \"db0b2703-500b-47e2-9165-8808feae0778\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3faec0-95db6ff75-w9729" Sep 11 20:24:45.311031 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:24:45.310906 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db0b2703-500b-47e2-9165-8808feae0778-openshift-service-ca\") pod \"maas-api-e2e-api-a-3faec0-95db6ff75-w9729\" (UID: \"db0b2703-500b-47e2-9165-8808feae0778\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3faec0-95db6ff75-w9729" Sep 11 20:24:45.311147 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:24:45.311032 2546 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-3faec0-serving-cert: secret "maas-api-e2e-api-a-3faec0-serving-cert" not found Sep 11 20:24:45.311147 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:24:45.311108 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db0b2703-500b-47e2-9165-8808feae0778-maas-api-tls podName:db0b2703-500b-47e2-9165-8808feae0778 nodeName:}" failed. No retries permitted until 2026-09-11 20:24:45.811092048 +0000 UTC m=+1455.859142370 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/db0b2703-500b-47e2-9165-8808feae0778-maas-api-tls") pod "maas-api-e2e-api-a-3faec0-95db6ff75-w9729" (UID: "db0b2703-500b-47e2-9165-8808feae0778") : secret "maas-api-e2e-api-a-3faec0-serving-cert" not found Sep 11 20:24:45.311572 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:24:45.311549 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db0b2703-500b-47e2-9165-8808feae0778-openshift-service-ca\") pod \"maas-api-e2e-api-a-3faec0-95db6ff75-w9729\" (UID: \"db0b2703-500b-47e2-9165-8808feae0778\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3faec0-95db6ff75-w9729" Sep 11 20:24:45.313142 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:24:45.313123 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db0b2703-500b-47e2-9165-8808feae0778-metrics-certs\") pod \"maas-api-e2e-api-a-3faec0-95db6ff75-w9729\" (UID: \"db0b2703-500b-47e2-9165-8808feae0778\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3faec0-95db6ff75-w9729" Sep 11 20:24:45.320253 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:24:45.320234 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkr5r\" (UniqueName: \"kubernetes.io/projected/db0b2703-500b-47e2-9165-8808feae0778-kube-api-access-fkr5r\") pod \"maas-api-e2e-api-a-3faec0-95db6ff75-w9729\" (UID: \"db0b2703-500b-47e2-9165-8808feae0778\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3faec0-95db6ff75-w9729" Sep 11 20:24:45.547598 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:24:45.547538 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:24:45.752567 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:24:45.752532 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:24:45.817193 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:24:45.817158 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db0b2703-500b-47e2-9165-8808feae0778-maas-api-tls\") pod \"maas-api-e2e-api-a-3faec0-95db6ff75-w9729\" (UID: \"db0b2703-500b-47e2-9165-8808feae0778\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3faec0-95db6ff75-w9729" Sep 11 20:24:45.819529 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:24:45.819498 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db0b2703-500b-47e2-9165-8808feae0778-maas-api-tls\") pod \"maas-api-e2e-api-a-3faec0-95db6ff75-w9729\" (UID: \"db0b2703-500b-47e2-9165-8808feae0778\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3faec0-95db6ff75-w9729" Sep 11 20:24:45.822952 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:24:45.822926 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:24:45.955771 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:24:45.955735 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3faec0-95db6ff75-w9729" Sep 11 20:24:46.234832 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:24:46.234800 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3faec0-95db6ff75-w9729"] Sep 11 20:24:46.236590 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:24:46.236546 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb0b2703_500b_47e2_9165_8808feae0778.slice/crio-0c72cc6b57ea554bbb5e86cd68c966d79205857a5fb7d51055ffba48c1a8a9d9 WatchSource:0}: Error finding container 0c72cc6b57ea554bbb5e86cd68c966d79205857a5fb7d51055ffba48c1a8a9d9: Status 404 returned error can't find the container with id 0c72cc6b57ea554bbb5e86cd68c966d79205857a5fb7d51055ffba48c1a8a9d9 Sep 11 20:24:46.237964 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:24:46.237948 2546 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 20:24:46.294473 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:24:46.294397 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3faec0-95db6ff75-w9729" event={"ID":"db0b2703-500b-47e2-9165-8808feae0778","Type":"ContainerStarted","Data":"0c72cc6b57ea554bbb5e86cd68c966d79205857a5fb7d51055ffba48c1a8a9d9"} Sep 11 20:24:47.301281 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:24:47.301191 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3faec0-95db6ff75-w9729" event={"ID":"db0b2703-500b-47e2-9165-8808feae0778","Type":"ContainerStarted","Data":"a8a26a88cccf02b84185d3e69810df54b0ff2df9e827dfeb437803d5432814cc"} Sep 11 20:24:47.301281 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:24:47.301253 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3faec0-95db6ff75-w9729" Sep 11 20:24:47.323658 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:24:47.323613 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3faec0-95db6ff75-w9729" podStartSLOduration=1.553715457 podStartE2EDuration="2.323599239s" totalImagesPullingTime="769.883782ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:24:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:24:47.321260107 +0000 UTC m=+1457.369310464" watchObservedRunningTime="2026-09-11 20:24:47.323599239 +0000 UTC m=+1457.371649584" Sep 11 20:24:48.076546 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:24:48.076489 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:24:53.311668 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:24:53.311637 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3faec0-95db6ff75-w9729" Sep 11 20:25:08.209360 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:08.209313 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:25:08.311269 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:08.311236 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:25:09.118207 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:09.118176 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:25:09.799006 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:09.798976 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:25:10.504362 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:10.504327 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:25:10.778026 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:10.777945 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:25:11.380038 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:11.379999 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:25:11.485477 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:11.485445 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:25:11.664984 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:11.664898 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:25:11.763865 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:11.763836 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:25:12.285562 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:12.285522 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:25:13.684912 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:13.684859 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:25:14.209219 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:14.209180 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:25:14.864067 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:14.864029 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:25:14.956484 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:14.956453 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:25:15.057520 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:15.057485 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:25:16.024752 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:16.024722 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:25:16.828819 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:16.828785 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:25:17.720458 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:17.720419 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:25:18.618591 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:18.618550 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:25:19.304835 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:19.304799 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:25:20.080143 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:20.080101 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b0d28c-f6dc9879c-h8c2h"] Sep 11 20:25:20.085834 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:20.085807 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b0d28c-f6dc9879c-h8c2h" Sep 11 20:25:20.087801 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:20.087779 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-b0d28c-serving-cert\"" Sep 11 20:25:20.095205 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:20.095176 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b0d28c-f6dc9879c-h8c2h"] Sep 11 20:25:20.107466 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:20.107439 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:25:20.162249 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:20.162221 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026-metrics-certs\") pod \"maas-api-e2e-api-b-b0d28c-f6dc9879c-h8c2h\" (UID: \"b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b0d28c-f6dc9879c-h8c2h" Sep 11 20:25:20.162408 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:20.162298 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026-maas-api-tls\") pod \"maas-api-e2e-api-b-b0d28c-f6dc9879c-h8c2h\" (UID: \"b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b0d28c-f6dc9879c-h8c2h" Sep 11 20:25:20.162408 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:20.162339 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026-openshift-service-ca\") pod \"maas-api-e2e-api-b-b0d28c-f6dc9879c-h8c2h\" (UID: \"b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b0d28c-f6dc9879c-h8c2h" Sep 11 20:25:20.162408 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:20.162392 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66kg5\" (UniqueName: \"kubernetes.io/projected/b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026-kube-api-access-66kg5\") pod \"maas-api-e2e-api-b-b0d28c-f6dc9879c-h8c2h\" (UID: \"b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b0d28c-f6dc9879c-h8c2h" Sep 11 20:25:20.207556 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:20.207524 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:25:20.263854 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:20.263819 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026-metrics-certs\") pod \"maas-api-e2e-api-b-b0d28c-f6dc9879c-h8c2h\" (UID: \"b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b0d28c-f6dc9879c-h8c2h" Sep 11 20:25:20.263995 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:20.263902 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026-maas-api-tls\") pod \"maas-api-e2e-api-b-b0d28c-f6dc9879c-h8c2h\" (UID: \"b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b0d28c-f6dc9879c-h8c2h" Sep 11 20:25:20.263995 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:20.263938 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026-openshift-service-ca\") pod \"maas-api-e2e-api-b-b0d28c-f6dc9879c-h8c2h\" (UID: \"b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b0d28c-f6dc9879c-h8c2h" Sep 11 20:25:20.263995 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:20.263973 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66kg5\" (UniqueName: \"kubernetes.io/projected/b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026-kube-api-access-66kg5\") pod \"maas-api-e2e-api-b-b0d28c-f6dc9879c-h8c2h\" (UID: \"b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b0d28c-f6dc9879c-h8c2h" Sep 11 20:25:20.264116 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:25:20.264036 2546 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-b0d28c-serving-cert: secret "maas-api-e2e-api-b-b0d28c-serving-cert" not found Sep 11 20:25:20.264151 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:25:20.264115 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026-maas-api-tls podName:b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026 nodeName:}" failed. No retries permitted until 2026-09-11 20:25:20.764093317 +0000 UTC m=+1490.812143658 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026-maas-api-tls") pod "maas-api-e2e-api-b-b0d28c-f6dc9879c-h8c2h" (UID: "b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026") : secret "maas-api-e2e-api-b-b0d28c-serving-cert" not found Sep 11 20:25:20.264613 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:20.264567 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026-openshift-service-ca\") pod \"maas-api-e2e-api-b-b0d28c-f6dc9879c-h8c2h\" (UID: \"b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b0d28c-f6dc9879c-h8c2h" Sep 11 20:25:20.266277 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:20.266260 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026-metrics-certs\") pod \"maas-api-e2e-api-b-b0d28c-f6dc9879c-h8c2h\" (UID: \"b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b0d28c-f6dc9879c-h8c2h" Sep 11 20:25:20.271360 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:20.271335 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66kg5\" (UniqueName: \"kubernetes.io/projected/b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026-kube-api-access-66kg5\") pod \"maas-api-e2e-api-b-b0d28c-f6dc9879c-h8c2h\" (UID: \"b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b0d28c-f6dc9879c-h8c2h" Sep 11 20:25:20.559564 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:20.559517 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:25:20.688511 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:20.688480 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:25:20.769055 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:20.769024 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026-maas-api-tls\") pod \"maas-api-e2e-api-b-b0d28c-f6dc9879c-h8c2h\" (UID: \"b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b0d28c-f6dc9879c-h8c2h" Sep 11 20:25:20.771504 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:20.771478 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026-maas-api-tls\") pod \"maas-api-e2e-api-b-b0d28c-f6dc9879c-h8c2h\" (UID: \"b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b0d28c-f6dc9879c-h8c2h" Sep 11 20:25:20.790630 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:20.790604 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:25:20.866351 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:20.866318 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b0d28c-76dc4cd778-svf5d"] Sep 11 20:25:20.870283 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:20.870262 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b0d28c-76dc4cd778-svf5d" Sep 11 20:25:20.872296 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:20.872277 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-b0d28c-dockercfg-6nhpq\"" Sep 11 20:25:20.872424 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:20.872299 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-b0d28c\"" Sep 11 20:25:20.883624 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:20.883594 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b0d28c-76dc4cd778-svf5d"] Sep 11 20:25:20.971136 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:20.971095 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ad46135-0cc6-41f1-8e00-6a6f58f2be20-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b0d28c-76dc4cd778-svf5d\" (UID: \"9ad46135-0cc6-41f1-8e00-6a6f58f2be20\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b0d28c-76dc4cd778-svf5d" Sep 11 20:25:20.971312 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:20.971152 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b5jt\" (UniqueName: \"kubernetes.io/projected/9ad46135-0cc6-41f1-8e00-6a6f58f2be20-kube-api-access-4b5jt\") pod \"payload-pre-processing-e2e-api-b-b0d28c-76dc4cd778-svf5d\" (UID: \"9ad46135-0cc6-41f1-8e00-6a6f58f2be20\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b0d28c-76dc4cd778-svf5d" Sep 11 20:25:20.999870 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:20.999829 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b0d28c-f6dc9879c-h8c2h" Sep 11 20:25:21.072326 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:21.072298 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ad46135-0cc6-41f1-8e00-6a6f58f2be20-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b0d28c-76dc4cd778-svf5d\" (UID: \"9ad46135-0cc6-41f1-8e00-6a6f58f2be20\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b0d28c-76dc4cd778-svf5d" Sep 11 20:25:21.072473 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:21.072334 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b5jt\" (UniqueName: \"kubernetes.io/projected/9ad46135-0cc6-41f1-8e00-6a6f58f2be20-kube-api-access-4b5jt\") pod \"payload-pre-processing-e2e-api-b-b0d28c-76dc4cd778-svf5d\" (UID: \"9ad46135-0cc6-41f1-8e00-6a6f58f2be20\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b0d28c-76dc4cd778-svf5d" Sep 11 20:25:21.073150 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:21.073122 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ad46135-0cc6-41f1-8e00-6a6f58f2be20-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b0d28c-76dc4cd778-svf5d\" (UID: \"9ad46135-0cc6-41f1-8e00-6a6f58f2be20\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b0d28c-76dc4cd778-svf5d" Sep 11 20:25:21.082650 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:21.082625 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b5jt\" (UniqueName: \"kubernetes.io/projected/9ad46135-0cc6-41f1-8e00-6a6f58f2be20-kube-api-access-4b5jt\") pod \"payload-pre-processing-e2e-api-b-b0d28c-76dc4cd778-svf5d\" (UID: \"9ad46135-0cc6-41f1-8e00-6a6f58f2be20\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b0d28c-76dc4cd778-svf5d" Sep 11 20:25:21.183480 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:21.183400 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b0d28c-76dc4cd778-svf5d" Sep 11 20:25:21.275700 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:21.275668 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b0d28c-f6dc9879c-h8c2h"] Sep 11 20:25:21.277762 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:25:21.277734 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9efbdb9_9b8b_48b1_a3ec_1e85c5eab026.slice/crio-3746e0558db259582275f78dd7d7fc7c6a797de6fd30a4217b354fbdfc68b856 WatchSource:0}: Error finding container 3746e0558db259582275f78dd7d7fc7c6a797de6fd30a4217b354fbdfc68b856: Status 404 returned error can't find the container with id 3746e0558db259582275f78dd7d7fc7c6a797de6fd30a4217b354fbdfc68b856 Sep 11 20:25:21.458560 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:21.458537 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b0d28c-76dc4cd778-svf5d"] Sep 11 20:25:21.460820 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:25:21.460795 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ad46135_0cc6_41f1_8e00_6a6f58f2be20.slice/crio-8ed91bcdfc128ed1a358134ca98c3f2bf95ac8a2c530c7f72339117e6ce061cc WatchSource:0}: Error finding container 8ed91bcdfc128ed1a358134ca98c3f2bf95ac8a2c530c7f72339117e6ce061cc: Status 404 returned error can't find the container with id 8ed91bcdfc128ed1a358134ca98c3f2bf95ac8a2c530c7f72339117e6ce061cc Sep 11 20:25:21.495235 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:21.495199 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b0d28c-76dc4cd778-svf5d" event={"ID":"9ad46135-0cc6-41f1-8e00-6a6f58f2be20","Type":"ContainerStarted","Data":"8ed91bcdfc128ed1a358134ca98c3f2bf95ac8a2c530c7f72339117e6ce061cc"} Sep 11 20:25:21.496424 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:21.496400 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b0d28c-f6dc9879c-h8c2h" event={"ID":"b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026","Type":"ContainerStarted","Data":"3746e0558db259582275f78dd7d7fc7c6a797de6fd30a4217b354fbdfc68b856"} Sep 11 20:25:22.504393 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:22.504355 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b0d28c-76dc4cd778-svf5d" event={"ID":"9ad46135-0cc6-41f1-8e00-6a6f58f2be20","Type":"ContainerStarted","Data":"542f5c778f846220d65eb32a1cca36372c271a980d7a0c0449374e4aadf14747"} Sep 11 20:25:22.504869 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:22.504423 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b0d28c-76dc4cd778-svf5d" Sep 11 20:25:22.506060 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:22.506037 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b0d28c-f6dc9879c-h8c2h" event={"ID":"b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026","Type":"ContainerStarted","Data":"e2bc03e75ec9a50b57d08768dfde8f13a118bfe177f7c50618d020c2483ede13"} Sep 11 20:25:22.506181 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:22.506152 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b0d28c-f6dc9879c-h8c2h" Sep 11 20:25:22.523337 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:22.523286 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b0d28c-76dc4cd778-svf5d" podStartSLOduration=2.523272068 podStartE2EDuration="2.523272068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:25:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:25:22.520746818 +0000 UTC m=+1492.568797162" watchObservedRunningTime="2026-09-11 20:25:22.523272068 +0000 UTC m=+1492.571322412" Sep 11 20:25:22.537337 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:22.537287 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b0d28c-f6dc9879c-h8c2h" podStartSLOduration=2.218212134 podStartE2EDuration="2.537273255s" totalImagesPullingTime="319.061121ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:25:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:25:22.53565058 +0000 UTC m=+1492.583700956" watchObservedRunningTime="2026-09-11 20:25:22.537273255 +0000 UTC m=+1492.585323598" Sep 11 20:25:28.517453 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:28.517421 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b0d28c-f6dc9879c-h8c2h" Sep 11 20:25:33.514397 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:33.514366 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b0d28c-76dc4cd778-svf5d" Sep 11 20:25:37.430613 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:37.430555 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:25:41.131539 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:41.131507 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:25:41.736396 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:41.736352 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:25:42.231021 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:42.230988 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:25:43.129528 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:43.129496 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:25:43.986632 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:43.986601 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:25:44.722764 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:44.722729 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:25:44.943859 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:44.943830 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:25:45.533353 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:45.533322 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:25:46.266417 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:46.266389 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:25:46.834093 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:46.834062 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:25:47.307847 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:47.307815 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:25:48.320606 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:48.320562 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:25:49.121187 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:49.121158 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:25:50.115150 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:50.115119 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:25:51.121623 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:51.121590 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:25:52.027129 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:52.027092 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:25:52.869236 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:52.869199 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:25:53.636466 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:53.636431 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:25:54.539262 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:25:54.539233 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:00.178719 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:00.178688 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-22fc84-7fbff7d55d-zhngj"] Sep 11 20:26:00.179268 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:00.178926 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-22fc84-7fbff7d55d-zhngj" podUID="2647260f-49fc-450a-9e4b-fda58dfffddc" containerName="maas-api" containerID="cri-o://14ffec2bce9d656fcc9d3be80234e311428fa15c559502bd54799d14735fd252" gracePeriod=30 Sep 11 20:26:00.219974 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:00.219945 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:00.319761 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:00.319729 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:00.421656 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:00.421623 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:00.523372 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:00.521944 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:00.668915 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:00.668887 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-22fc84-7fbff7d55d-zhngj" Sep 11 20:26:00.728923 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:00.728828 2546 generic.go:356] "Generic (PLEG): container finished" podID="2647260f-49fc-450a-9e4b-fda58dfffddc" containerID="14ffec2bce9d656fcc9d3be80234e311428fa15c559502bd54799d14735fd252" exitCode=0 Sep 11 20:26:00.729085 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:00.728931 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-22fc84-7fbff7d55d-zhngj" event={"ID":"2647260f-49fc-450a-9e4b-fda58dfffddc","Type":"ContainerDied","Data":"14ffec2bce9d656fcc9d3be80234e311428fa15c559502bd54799d14735fd252"} Sep 11 20:26:00.729085 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:00.728979 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-22fc84-7fbff7d55d-zhngj" event={"ID":"2647260f-49fc-450a-9e4b-fda58dfffddc","Type":"ContainerDied","Data":"0bcd166e80b3719b155209e9217cc681ee1fb8d4b7d27cc553a21d6f319281b6"} Sep 11 20:26:00.729085 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:00.729004 2546 scope.go:117] "RemoveContainer" containerID="14ffec2bce9d656fcc9d3be80234e311428fa15c559502bd54799d14735fd252" Sep 11 20:26:00.729264 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:00.729194 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-22fc84-7fbff7d55d-zhngj" Sep 11 20:26:00.740395 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:00.740369 2546 scope.go:117] "RemoveContainer" containerID="14ffec2bce9d656fcc9d3be80234e311428fa15c559502bd54799d14735fd252" Sep 11 20:26:00.740712 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:26:00.740688 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14ffec2bce9d656fcc9d3be80234e311428fa15c559502bd54799d14735fd252\": container with ID starting with 14ffec2bce9d656fcc9d3be80234e311428fa15c559502bd54799d14735fd252 not found: ID does not exist" containerID="14ffec2bce9d656fcc9d3be80234e311428fa15c559502bd54799d14735fd252" Sep 11 20:26:00.740804 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:00.740724 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14ffec2bce9d656fcc9d3be80234e311428fa15c559502bd54799d14735fd252"} err="failed to get container status \"14ffec2bce9d656fcc9d3be80234e311428fa15c559502bd54799d14735fd252\": rpc error: code = NotFound desc = could not find container \"14ffec2bce9d656fcc9d3be80234e311428fa15c559502bd54799d14735fd252\": container with ID starting with 14ffec2bce9d656fcc9d3be80234e311428fa15c559502bd54799d14735fd252 not found: ID does not exist" Sep 11 20:26:00.788195 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:00.788174 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2647260f-49fc-450a-9e4b-fda58dfffddc-maas-api-tls\") pod \"2647260f-49fc-450a-9e4b-fda58dfffddc\" (UID: \"2647260f-49fc-450a-9e4b-fda58dfffddc\") " Sep 11 20:26:00.788319 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:00.788229 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2647260f-49fc-450a-9e4b-fda58dfffddc-metrics-certs\") pod \"2647260f-49fc-450a-9e4b-fda58dfffddc\" (UID: \"2647260f-49fc-450a-9e4b-fda58dfffddc\") " Sep 11 20:26:00.788319 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:00.788282 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hjw5t\" (UniqueName: \"kubernetes.io/projected/2647260f-49fc-450a-9e4b-fda58dfffddc-kube-api-access-hjw5t\") pod \"2647260f-49fc-450a-9e4b-fda58dfffddc\" (UID: \"2647260f-49fc-450a-9e4b-fda58dfffddc\") " Sep 11 20:26:00.788439 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:00.788361 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2647260f-49fc-450a-9e4b-fda58dfffddc-openshift-service-ca\") pod \"2647260f-49fc-450a-9e4b-fda58dfffddc\" (UID: \"2647260f-49fc-450a-9e4b-fda58dfffddc\") " Sep 11 20:26:00.788758 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:00.788733 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2647260f-49fc-450a-9e4b-fda58dfffddc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2647260f-49fc-450a-9e4b-fda58dfffddc" (UID: "2647260f-49fc-450a-9e4b-fda58dfffddc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:26:00.790174 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:00.790149 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2647260f-49fc-450a-9e4b-fda58dfffddc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2647260f-49fc-450a-9e4b-fda58dfffddc" (UID: "2647260f-49fc-450a-9e4b-fda58dfffddc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 20:26:00.790358 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:00.790331 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2647260f-49fc-450a-9e4b-fda58dfffddc-kube-api-access-hjw5t" (OuterVolumeSpecName: "kube-api-access-hjw5t") pod "2647260f-49fc-450a-9e4b-fda58dfffddc" (UID: "2647260f-49fc-450a-9e4b-fda58dfffddc"). InnerVolumeSpecName "kube-api-access-hjw5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:26:00.790458 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:00.790372 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2647260f-49fc-450a-9e4b-fda58dfffddc-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2647260f-49fc-450a-9e4b-fda58dfffddc" (UID: "2647260f-49fc-450a-9e4b-fda58dfffddc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 20:26:00.810936 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:00.810915 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:00.889370 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:00.889338 2546 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2647260f-49fc-450a-9e4b-fda58dfffddc-metrics-certs\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:26:00.889370 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:00.889366 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjw5t\" (UniqueName: \"kubernetes.io/projected/2647260f-49fc-450a-9e4b-fda58dfffddc-kube-api-access-hjw5t\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:26:00.889370 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:00.889376 2546 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2647260f-49fc-450a-9e4b-fda58dfffddc-openshift-service-ca\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:26:00.889621 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:00.889385 2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2647260f-49fc-450a-9e4b-fda58dfffddc-maas-api-tls\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:26:00.934313 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:00.934285 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:01.062058 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:01.062025 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-22fc84-7fbff7d55d-zhngj"] Sep 11 20:26:01.068917 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:01.068892 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-22fc84-7fbff7d55d-zhngj"] Sep 11 20:26:02.062788 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:02.062762 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:02.451240 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:02.451204 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2647260f-49fc-450a-9e4b-fda58dfffddc" path="/var/lib/kubelet/pods/2647260f-49fc-450a-9e4b-fda58dfffddc/volumes" Sep 11 20:26:02.787918 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:02.787838 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3faec0-95db6ff75-w9729"] Sep 11 20:26:02.788126 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:02.788080 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3faec0-95db6ff75-w9729" podUID="db0b2703-500b-47e2-9165-8808feae0778" containerName="maas-api" containerID="cri-o://a8a26a88cccf02b84185d3e69810df54b0ff2df9e827dfeb437803d5432814cc" gracePeriod=30 Sep 11 20:26:02.833597 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:02.833545 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:02.953886 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:02.953855 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:03.049841 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:03.049768 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:03.052665 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:03.052636 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3faec0-95db6ff75-w9729" Sep 11 20:26:03.111044 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:03.111013 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db0b2703-500b-47e2-9165-8808feae0778-openshift-service-ca\") pod \"db0b2703-500b-47e2-9165-8808feae0778\" (UID: \"db0b2703-500b-47e2-9165-8808feae0778\") " Sep 11 20:26:03.111451 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:03.111096 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fkr5r\" (UniqueName: \"kubernetes.io/projected/db0b2703-500b-47e2-9165-8808feae0778-kube-api-access-fkr5r\") pod \"db0b2703-500b-47e2-9165-8808feae0778\" (UID: \"db0b2703-500b-47e2-9165-8808feae0778\") " Sep 11 20:26:03.111451 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:03.111143 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db0b2703-500b-47e2-9165-8808feae0778-metrics-certs\") pod \"db0b2703-500b-47e2-9165-8808feae0778\" (UID: \"db0b2703-500b-47e2-9165-8808feae0778\") " Sep 11 20:26:03.111451 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:03.111351 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db0b2703-500b-47e2-9165-8808feae0778-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "db0b2703-500b-47e2-9165-8808feae0778" (UID: "db0b2703-500b-47e2-9165-8808feae0778"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:26:03.111568 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:03.111548 2546 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db0b2703-500b-47e2-9165-8808feae0778-openshift-service-ca\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:26:03.113122 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:03.113096 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db0b2703-500b-47e2-9165-8808feae0778-kube-api-access-fkr5r" (OuterVolumeSpecName: "kube-api-access-fkr5r") pod "db0b2703-500b-47e2-9165-8808feae0778" (UID: "db0b2703-500b-47e2-9165-8808feae0778"). InnerVolumeSpecName "kube-api-access-fkr5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:26:03.113223 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:03.113116 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db0b2703-500b-47e2-9165-8808feae0778-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "db0b2703-500b-47e2-9165-8808feae0778" (UID: "db0b2703-500b-47e2-9165-8808feae0778"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 20:26:03.156037 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:03.156010 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:03.212296 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:03.212263 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db0b2703-500b-47e2-9165-8808feae0778-maas-api-tls\") pod \"db0b2703-500b-47e2-9165-8808feae0778\" (UID: \"db0b2703-500b-47e2-9165-8808feae0778\") " Sep 11 20:26:03.212626 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:03.212606 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkr5r\" (UniqueName: \"kubernetes.io/projected/db0b2703-500b-47e2-9165-8808feae0778-kube-api-access-fkr5r\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:26:03.212626 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:03.212628 2546 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db0b2703-500b-47e2-9165-8808feae0778-metrics-certs\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:26:03.214242 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:03.214220 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db0b2703-500b-47e2-9165-8808feae0778-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "db0b2703-500b-47e2-9165-8808feae0778" (UID: "db0b2703-500b-47e2-9165-8808feae0778"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 20:26:03.254720 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:03.254695 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:03.314010 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:03.313916 2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db0b2703-500b-47e2-9165-8808feae0778-maas-api-tls\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:26:03.359081 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:03.359047 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:03.545268 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:03.545240 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:03.747115 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:03.747079 2546 generic.go:356] "Generic (PLEG): container finished" podID="db0b2703-500b-47e2-9165-8808feae0778" containerID="a8a26a88cccf02b84185d3e69810df54b0ff2df9e827dfeb437803d5432814cc" exitCode=0 Sep 11 20:26:03.747302 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:03.747154 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3faec0-95db6ff75-w9729" event={"ID":"db0b2703-500b-47e2-9165-8808feae0778","Type":"ContainerDied","Data":"a8a26a88cccf02b84185d3e69810df54b0ff2df9e827dfeb437803d5432814cc"} Sep 11 20:26:03.747302 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:03.747179 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3faec0-95db6ff75-w9729" Sep 11 20:26:03.747302 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:03.747193 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3faec0-95db6ff75-w9729" event={"ID":"db0b2703-500b-47e2-9165-8808feae0778","Type":"ContainerDied","Data":"0c72cc6b57ea554bbb5e86cd68c966d79205857a5fb7d51055ffba48c1a8a9d9"} Sep 11 20:26:03.747302 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:03.747213 2546 scope.go:117] "RemoveContainer" containerID="a8a26a88cccf02b84185d3e69810df54b0ff2df9e827dfeb437803d5432814cc" Sep 11 20:26:03.757339 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:03.757316 2546 scope.go:117] "RemoveContainer" containerID="a8a26a88cccf02b84185d3e69810df54b0ff2df9e827dfeb437803d5432814cc" Sep 11 20:26:03.757625 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:26:03.757605 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8a26a88cccf02b84185d3e69810df54b0ff2df9e827dfeb437803d5432814cc\": container with ID starting with a8a26a88cccf02b84185d3e69810df54b0ff2df9e827dfeb437803d5432814cc not found: ID does not exist" containerID="a8a26a88cccf02b84185d3e69810df54b0ff2df9e827dfeb437803d5432814cc" Sep 11 20:26:03.757694 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:03.757634 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8a26a88cccf02b84185d3e69810df54b0ff2df9e827dfeb437803d5432814cc"} err="failed to get container status \"a8a26a88cccf02b84185d3e69810df54b0ff2df9e827dfeb437803d5432814cc\": rpc error: code = NotFound desc = could not find container \"a8a26a88cccf02b84185d3e69810df54b0ff2df9e827dfeb437803d5432814cc\": container with ID starting with a8a26a88cccf02b84185d3e69810df54b0ff2df9e827dfeb437803d5432814cc not found: ID does not exist" Sep 11 20:26:03.769617 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:03.769592 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3faec0-95db6ff75-w9729"] Sep 11 20:26:03.773527 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:03.773506 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3faec0-95db6ff75-w9729"] Sep 11 20:26:04.185136 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:04.185102 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:04.451187 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:04.451102 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db0b2703-500b-47e2-9165-8808feae0778" path="/var/lib/kubelet/pods/db0b2703-500b-47e2-9165-8808feae0778/volumes" Sep 11 20:26:05.014743 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:05.014715 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:05.953170 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:05.953141 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:06.825223 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:06.825191 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:07.381407 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:07.381371 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:07.486679 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:07.486634 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:07.657923 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:07.657838 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:07.759360 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:07.759330 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:08.707491 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:08.707404 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:09.968986 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:09.968952 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:10.071680 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:10.071645 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:10.224995 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:10.224894 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:10.842621 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:10.842593 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:11.952218 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:11.952184 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:12.679314 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:12.679277 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:13.643972 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:13.643939 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:14.371323 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:14.371292 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:15.118618 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:15.118566 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:16.022165 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:16.022132 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:23.783499 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:23.783465 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:23.885439 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:23.885406 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:25.189226 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:25.189190 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:26.128305 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:26.128271 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b0d28c-f6dc9879c-h8c2h"] Sep 11 20:26:26.128703 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:26.128671 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b0d28c-f6dc9879c-h8c2h" podUID="b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026" containerName="maas-api" containerID="cri-o://e2bc03e75ec9a50b57d08768dfde8f13a118bfe177f7c50618d020c2483ede13" gracePeriod=30 Sep 11 20:26:26.158062 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:26.158016 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:26.196768 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:26.196705 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b0d28c-76dc4cd778-svf5d"] Sep 11 20:26:26.197124 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:26.196991 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b0d28c-76dc4cd778-svf5d" podUID="9ad46135-0cc6-41f1-8e00-6a6f58f2be20" containerName="payload-pre-processing" containerID="cri-o://542f5c778f846220d65eb32a1cca36372c271a980d7a0c0449374e4aadf14747" gracePeriod=30 Sep 11 20:26:26.252897 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:26.252865 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:26.353538 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:26.353485 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:26.420344 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:26.420308 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:26.483127 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:26.481934 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b0d28c-f6dc9879c-h8c2h" Sep 11 20:26:26.516726 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:26.516701 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b0d28c-76dc4cd778-svf5d" Sep 11 20:26:26.649399 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:26.649371 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026-maas-api-tls\") pod \"b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026\" (UID: \"b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026\") " Sep 11 20:26:26.649612 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:26.649407 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026-openshift-service-ca\") pod \"b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026\" (UID: \"b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026\") " Sep 11 20:26:26.649612 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:26.649437 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026-metrics-certs\") pod \"b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026\" (UID: \"b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026\") " Sep 11 20:26:26.649612 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:26.649456 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4b5jt\" (UniqueName: \"kubernetes.io/projected/9ad46135-0cc6-41f1-8e00-6a6f58f2be20-kube-api-access-4b5jt\") pod \"9ad46135-0cc6-41f1-8e00-6a6f58f2be20\" (UID: \"9ad46135-0cc6-41f1-8e00-6a6f58f2be20\") " Sep 11 20:26:26.649612 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:26.649594 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ad46135-0cc6-41f1-8e00-6a6f58f2be20-plugins-config-volume\") pod \"9ad46135-0cc6-41f1-8e00-6a6f58f2be20\" (UID: \"9ad46135-0cc6-41f1-8e00-6a6f58f2be20\") " Sep 11 20:26:26.649856 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:26.649628 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-66kg5\" (UniqueName: \"kubernetes.io/projected/b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026-kube-api-access-66kg5\") pod \"b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026\" (UID: \"b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026\") " Sep 11 20:26:26.649933 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:26.649830 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026" (UID: "b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:26:26.650064 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:26.650021 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ad46135-0cc6-41f1-8e00-6a6f58f2be20-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9ad46135-0cc6-41f1-8e00-6a6f58f2be20" (UID: "9ad46135-0cc6-41f1-8e00-6a6f58f2be20"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:26:26.650207 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:26.650079 2546 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026-openshift-service-ca\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:26:26.651707 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:26.651684 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026" (UID: "b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 20:26:26.651827 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:26.651718 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026" (UID: "b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 20:26:26.651827 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:26.651752 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ad46135-0cc6-41f1-8e00-6a6f58f2be20-kube-api-access-4b5jt" (OuterVolumeSpecName: "kube-api-access-4b5jt") pod "9ad46135-0cc6-41f1-8e00-6a6f58f2be20" (UID: "9ad46135-0cc6-41f1-8e00-6a6f58f2be20"). InnerVolumeSpecName "kube-api-access-4b5jt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:26:26.651827 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:26.651795 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026-kube-api-access-66kg5" (OuterVolumeSpecName: "kube-api-access-66kg5") pod "b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026" (UID: "b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026"). InnerVolumeSpecName "kube-api-access-66kg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:26:26.750774 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:26.750706 2546 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026-metrics-certs\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:26:26.750774 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:26.750731 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b5jt\" (UniqueName: \"kubernetes.io/projected/9ad46135-0cc6-41f1-8e00-6a6f58f2be20-kube-api-access-4b5jt\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:26:26.750774 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:26.750741 2546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ad46135-0cc6-41f1-8e00-6a6f58f2be20-plugins-config-volume\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:26:26.750774 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:26.750750 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66kg5\" (UniqueName: \"kubernetes.io/projected/b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026-kube-api-access-66kg5\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:26:26.750774 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:26.750758 2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026-maas-api-tls\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:26:26.772779 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:26.772747 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:26.873173 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:26.873136 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:26.880212 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:26.880184 2546 generic.go:356] "Generic (PLEG): container finished" podID="9ad46135-0cc6-41f1-8e00-6a6f58f2be20" containerID="542f5c778f846220d65eb32a1cca36372c271a980d7a0c0449374e4aadf14747" exitCode=0 Sep 11 20:26:26.880368 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:26.880229 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b0d28c-76dc4cd778-svf5d" Sep 11 20:26:26.880368 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:26.880222 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b0d28c-76dc4cd778-svf5d" event={"ID":"9ad46135-0cc6-41f1-8e00-6a6f58f2be20","Type":"ContainerDied","Data":"542f5c778f846220d65eb32a1cca36372c271a980d7a0c0449374e4aadf14747"} Sep 11 20:26:26.880368 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:26.880292 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b0d28c-76dc4cd778-svf5d" event={"ID":"9ad46135-0cc6-41f1-8e00-6a6f58f2be20","Type":"ContainerDied","Data":"8ed91bcdfc128ed1a358134ca98c3f2bf95ac8a2c530c7f72339117e6ce061cc"} Sep 11 20:26:26.880368 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:26.880324 2546 scope.go:117] "RemoveContainer" containerID="542f5c778f846220d65eb32a1cca36372c271a980d7a0c0449374e4aadf14747" Sep 11 20:26:26.881682 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:26.881662 2546 generic.go:356] "Generic (PLEG): container finished" podID="b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026" containerID="e2bc03e75ec9a50b57d08768dfde8f13a118bfe177f7c50618d020c2483ede13" exitCode=0 Sep 11 20:26:26.881804 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:26.881684 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b0d28c-f6dc9879c-h8c2h" event={"ID":"b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026","Type":"ContainerDied","Data":"e2bc03e75ec9a50b57d08768dfde8f13a118bfe177f7c50618d020c2483ede13"} Sep 11 20:26:26.881804 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:26.881721 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b0d28c-f6dc9879c-h8c2h" Sep 11 20:26:26.881804 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:26.881724 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b0d28c-f6dc9879c-h8c2h" event={"ID":"b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026","Type":"ContainerDied","Data":"3746e0558db259582275f78dd7d7fc7c6a797de6fd30a4217b354fbdfc68b856"} Sep 11 20:26:26.891131 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:26.891113 2546 scope.go:117] "RemoveContainer" containerID="542f5c778f846220d65eb32a1cca36372c271a980d7a0c0449374e4aadf14747" Sep 11 20:26:26.891397 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:26:26.891375 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"542f5c778f846220d65eb32a1cca36372c271a980d7a0c0449374e4aadf14747\": container with ID starting with 542f5c778f846220d65eb32a1cca36372c271a980d7a0c0449374e4aadf14747 not found: ID does not exist" containerID="542f5c778f846220d65eb32a1cca36372c271a980d7a0c0449374e4aadf14747" Sep 11 20:26:26.891450 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:26.891410 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"542f5c778f846220d65eb32a1cca36372c271a980d7a0c0449374e4aadf14747"} err="failed to get container status \"542f5c778f846220d65eb32a1cca36372c271a980d7a0c0449374e4aadf14747\": rpc error: code = NotFound desc = could not find container \"542f5c778f846220d65eb32a1cca36372c271a980d7a0c0449374e4aadf14747\": container with ID starting with 542f5c778f846220d65eb32a1cca36372c271a980d7a0c0449374e4aadf14747 not found: ID does not exist" Sep 11 20:26:26.891450 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:26.891432 2546 scope.go:117] "RemoveContainer" containerID="e2bc03e75ec9a50b57d08768dfde8f13a118bfe177f7c50618d020c2483ede13" Sep 11 20:26:26.905022 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:26.904526 2546 scope.go:117] "RemoveContainer" containerID="e2bc03e75ec9a50b57d08768dfde8f13a118bfe177f7c50618d020c2483ede13" Sep 11 20:26:26.905598 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:26:26.905499 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2bc03e75ec9a50b57d08768dfde8f13a118bfe177f7c50618d020c2483ede13\": container with ID starting with e2bc03e75ec9a50b57d08768dfde8f13a118bfe177f7c50618d020c2483ede13 not found: ID does not exist" containerID="e2bc03e75ec9a50b57d08768dfde8f13a118bfe177f7c50618d020c2483ede13" Sep 11 20:26:26.905598 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:26.905539 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2bc03e75ec9a50b57d08768dfde8f13a118bfe177f7c50618d020c2483ede13"} err="failed to get container status \"e2bc03e75ec9a50b57d08768dfde8f13a118bfe177f7c50618d020c2483ede13\": rpc error: code = NotFound desc = could not find container \"e2bc03e75ec9a50b57d08768dfde8f13a118bfe177f7c50618d020c2483ede13\": container with ID starting with e2bc03e75ec9a50b57d08768dfde8f13a118bfe177f7c50618d020c2483ede13 not found: ID does not exist" Sep 11 20:26:26.914326 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:26.914298 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b0d28c-76dc4cd778-svf5d"] Sep 11 20:26:26.919098 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:26.919075 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b0d28c-76dc4cd778-svf5d"] Sep 11 20:26:26.932259 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:26.932238 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b0d28c-f6dc9879c-h8c2h"] Sep 11 20:26:26.935309 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:26.935282 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b0d28c-f6dc9879c-h8c2h"] Sep 11 20:26:27.620645 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:27.620610 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:28.451925 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:28.451898 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ad46135-0cc6-41f1-8e00-6a6f58f2be20" path="/var/lib/kubelet/pods/9ad46135-0cc6-41f1-8e00-6a6f58f2be20/volumes" Sep 11 20:26:28.452325 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:28.452311 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026" path="/var/lib/kubelet/pods/b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026/volumes" Sep 11 20:26:28.505521 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:28.505495 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:29.276683 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:29.276647 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:30.272078 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:30.272038 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:31.391176 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:31.391144 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:33.076615 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:33.076569 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:33.212919 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:33.212883 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:33.484938 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:33.484900 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:34.825081 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:34.825048 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:35.750545 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:35.750502 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:36.688092 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:36.688064 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:37.762611 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:37.762490 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:38.865846 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:38.865806 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:39.798811 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:39.798782 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:43.673213 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:43.673180 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:43.999215 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:43.999132 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:44.798199 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:44.798164 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ea327d-7c65697774-l6rzd"] Sep 11 20:26:44.799127 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:44.799094 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ea327d-7c65697774-l6rzd" podUID="dc977a98-1ebc-44cf-8eb0-6f6693b092d3" containerName="maas-api" containerID="cri-o://35a82174331b01a8f1c06cade1411877cf5426a6cfd8967901ff8e38f39b29ee" gracePeriod=30 Sep 11 20:26:44.823794 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:44.823757 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:44.868871 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:44.868835 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-ea327d-7999bdbc7-xpxc5"] Sep 11 20:26:44.869085 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:44.869061 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ea327d-7999bdbc7-xpxc5" podUID="3abcbcd7-5ecd-417c-a2c2-378966ad14fd" containerName="payload-processing" containerID="cri-o://3b13f7e96f114cc39a97e2260ec0622489092095d6cffade34c8d45644e3ea61" gracePeriod=30 Sep 11 20:26:44.880544 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:44.880490 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ea327d-6f44c946c8-ccphp"] Sep 11 20:26:44.880996 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:44.880969 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ea327d-6f44c946c8-ccphp" podUID="b7301644-7f5a-4fb0-a7f3-908f028c938c" containerName="payload-pre-processing" containerID="cri-o://a521b6048cebb7aba1b156874051b0937749211ab0857a18e2b50bb8fefb17ff" gracePeriod=30 Sep 11 20:26:44.921181 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:44.921139 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:44.932492 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:26:44.932456 2546 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7301644_7f5a_4fb0_a7f3_908f028c938c.slice/crio-a521b6048cebb7aba1b156874051b0937749211ab0857a18e2b50bb8fefb17ff.scope\": RecentStats: unable to find data in memory cache]" Sep 11 20:26:44.984755 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:44.984727 2546 generic.go:356] "Generic (PLEG): container finished" podID="3abcbcd7-5ecd-417c-a2c2-378966ad14fd" containerID="3b13f7e96f114cc39a97e2260ec0622489092095d6cffade34c8d45644e3ea61" exitCode=0 Sep 11 20:26:44.984910 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:44.984796 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ea327d-7999bdbc7-xpxc5" event={"ID":"3abcbcd7-5ecd-417c-a2c2-378966ad14fd","Type":"ContainerDied","Data":"3b13f7e96f114cc39a97e2260ec0622489092095d6cffade34c8d45644e3ea61"} Sep 11 20:26:44.986767 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:44.986738 2546 generic.go:356] "Generic (PLEG): container finished" podID="dc977a98-1ebc-44cf-8eb0-6f6693b092d3" containerID="35a82174331b01a8f1c06cade1411877cf5426a6cfd8967901ff8e38f39b29ee" exitCode=0 Sep 11 20:26:44.986887 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:44.986809 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ea327d-7c65697774-l6rzd" event={"ID":"dc977a98-1ebc-44cf-8eb0-6f6693b092d3","Type":"ContainerDied","Data":"35a82174331b01a8f1c06cade1411877cf5426a6cfd8967901ff8e38f39b29ee"} Sep 11 20:26:44.994245 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:44.994151 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:45.093412 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:45.093324 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:45.172486 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:45.172402 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:45.172836 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:45.172754 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ea327d-7999bdbc7-xpxc5" Sep 11 20:26:45.188054 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:45.188026 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ea327d-6f44c946c8-ccphp" Sep 11 20:26:45.200375 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:45.200345 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ea327d-7c65697774-l6rzd" Sep 11 20:26:45.224039 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:45.224013 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc977a98-1ebc-44cf-8eb0-6f6693b092d3-metrics-certs\") pod \"dc977a98-1ebc-44cf-8eb0-6f6693b092d3\" (UID: \"dc977a98-1ebc-44cf-8eb0-6f6693b092d3\") " Sep 11 20:26:45.224204 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:45.224055 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3abcbcd7-5ecd-417c-a2c2-378966ad14fd-plugins-config-volume\") pod \"3abcbcd7-5ecd-417c-a2c2-378966ad14fd\" (UID: \"3abcbcd7-5ecd-417c-a2c2-378966ad14fd\") " Sep 11 20:26:45.224204 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:45.224161 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9g9vk\" (UniqueName: \"kubernetes.io/projected/3abcbcd7-5ecd-417c-a2c2-378966ad14fd-kube-api-access-9g9vk\") pod \"3abcbcd7-5ecd-417c-a2c2-378966ad14fd\" (UID: \"3abcbcd7-5ecd-417c-a2c2-378966ad14fd\") " Sep 11 20:26:45.224328 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:45.224224 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc977a98-1ebc-44cf-8eb0-6f6693b092d3-maas-api-tls\") pod \"dc977a98-1ebc-44cf-8eb0-6f6693b092d3\" (UID: \"dc977a98-1ebc-44cf-8eb0-6f6693b092d3\") " Sep 11 20:26:45.224328 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:45.224277 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc977a98-1ebc-44cf-8eb0-6f6693b092d3-openshift-service-ca\") pod \"dc977a98-1ebc-44cf-8eb0-6f6693b092d3\" (UID: \"dc977a98-1ebc-44cf-8eb0-6f6693b092d3\") " Sep 11 20:26:45.224328 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:45.224307 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fkjtx\" (UniqueName: \"kubernetes.io/projected/dc977a98-1ebc-44cf-8eb0-6f6693b092d3-kube-api-access-fkjtx\") pod \"dc977a98-1ebc-44cf-8eb0-6f6693b092d3\" (UID: \"dc977a98-1ebc-44cf-8eb0-6f6693b092d3\") " Sep 11 20:26:45.224483 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:45.224346 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7301644-7f5a-4fb0-a7f3-908f028c938c-plugins-config-volume\") pod \"b7301644-7f5a-4fb0-a7f3-908f028c938c\" (UID: \"b7301644-7f5a-4fb0-a7f3-908f028c938c\") " Sep 11 20:26:45.224483 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:45.224397 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8bc4\" (UniqueName: \"kubernetes.io/projected/b7301644-7f5a-4fb0-a7f3-908f028c938c-kube-api-access-v8bc4\") pod \"b7301644-7f5a-4fb0-a7f3-908f028c938c\" (UID: \"b7301644-7f5a-4fb0-a7f3-908f028c938c\") " Sep 11 20:26:45.226233 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:45.225969 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7301644-7f5a-4fb0-a7f3-908f028c938c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b7301644-7f5a-4fb0-a7f3-908f028c938c" (UID: "b7301644-7f5a-4fb0-a7f3-908f028c938c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:26:45.226233 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:45.226171 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc977a98-1ebc-44cf-8eb0-6f6693b092d3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "dc977a98-1ebc-44cf-8eb0-6f6693b092d3" (UID: "dc977a98-1ebc-44cf-8eb0-6f6693b092d3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:26:45.226233 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:45.226195 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3abcbcd7-5ecd-417c-a2c2-378966ad14fd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3abcbcd7-5ecd-417c-a2c2-378966ad14fd" (UID: "3abcbcd7-5ecd-417c-a2c2-378966ad14fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:26:45.228901 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:45.228859 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc977a98-1ebc-44cf-8eb0-6f6693b092d3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "dc977a98-1ebc-44cf-8eb0-6f6693b092d3" (UID: "dc977a98-1ebc-44cf-8eb0-6f6693b092d3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 20:26:45.229617 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:45.229553 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc977a98-1ebc-44cf-8eb0-6f6693b092d3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "dc977a98-1ebc-44cf-8eb0-6f6693b092d3" (UID: "dc977a98-1ebc-44cf-8eb0-6f6693b092d3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 20:26:45.229722 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:45.229645 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7301644-7f5a-4fb0-a7f3-908f028c938c-kube-api-access-v8bc4" (OuterVolumeSpecName: "kube-api-access-v8bc4") pod "b7301644-7f5a-4fb0-a7f3-908f028c938c" (UID: "b7301644-7f5a-4fb0-a7f3-908f028c938c"). InnerVolumeSpecName "kube-api-access-v8bc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:26:45.230068 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:45.230046 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3abcbcd7-5ecd-417c-a2c2-378966ad14fd-kube-api-access-9g9vk" (OuterVolumeSpecName: "kube-api-access-9g9vk") pod "3abcbcd7-5ecd-417c-a2c2-378966ad14fd" (UID: "3abcbcd7-5ecd-417c-a2c2-378966ad14fd"). InnerVolumeSpecName "kube-api-access-9g9vk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:26:45.230783 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:45.230763 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc977a98-1ebc-44cf-8eb0-6f6693b092d3-kube-api-access-fkjtx" (OuterVolumeSpecName: "kube-api-access-fkjtx") pod "dc977a98-1ebc-44cf-8eb0-6f6693b092d3" (UID: "dc977a98-1ebc-44cf-8eb0-6f6693b092d3"). InnerVolumeSpecName "kube-api-access-fkjtx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:26:45.326113 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:45.326083 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkjtx\" (UniqueName: \"kubernetes.io/projected/dc977a98-1ebc-44cf-8eb0-6f6693b092d3-kube-api-access-fkjtx\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:26:45.326113 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:45.326109 2546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7301644-7f5a-4fb0-a7f3-908f028c938c-plugins-config-volume\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:26:45.326113 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:45.326119 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8bc4\" (UniqueName: \"kubernetes.io/projected/b7301644-7f5a-4fb0-a7f3-908f028c938c-kube-api-access-v8bc4\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:26:45.326393 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:45.326129 2546 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc977a98-1ebc-44cf-8eb0-6f6693b092d3-metrics-certs\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:26:45.326393 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:45.326142 2546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3abcbcd7-5ecd-417c-a2c2-378966ad14fd-plugins-config-volume\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:26:45.326393 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:45.326155 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g9vk\" (UniqueName: \"kubernetes.io/projected/3abcbcd7-5ecd-417c-a2c2-378966ad14fd-kube-api-access-9g9vk\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:26:45.326393 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:45.326164 2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc977a98-1ebc-44cf-8eb0-6f6693b092d3-maas-api-tls\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:26:45.326393 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:45.326176 2546 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc977a98-1ebc-44cf-8eb0-6f6693b092d3-openshift-service-ca\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:26:45.520017 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:45.519919 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:45.627024 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:45.626990 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:45.993646 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:45.993618 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ea327d-7999bdbc7-xpxc5" Sep 11 20:26:45.993646 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:45.993638 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ea327d-7999bdbc7-xpxc5" event={"ID":"3abcbcd7-5ecd-417c-a2c2-378966ad14fd","Type":"ContainerDied","Data":"cb104e87fb3438d69c967c34abe1704f9640c283748844ad037783cf63d2bc1e"} Sep 11 20:26:45.994185 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:45.993705 2546 scope.go:117] "RemoveContainer" containerID="3b13f7e96f114cc39a97e2260ec0622489092095d6cffade34c8d45644e3ea61" Sep 11 20:26:45.995153 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:45.995083 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ea327d-7c65697774-l6rzd" Sep 11 20:26:45.995153 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:45.995118 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ea327d-7c65697774-l6rzd" event={"ID":"dc977a98-1ebc-44cf-8eb0-6f6693b092d3","Type":"ContainerDied","Data":"c5e170179705069af29006aa02c044614167cb21365575c4e30b1b5d4a205c06"} Sep 11 20:26:45.996629 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:45.996602 2546 generic.go:356] "Generic (PLEG): container finished" podID="b7301644-7f5a-4fb0-a7f3-908f028c938c" containerID="a521b6048cebb7aba1b156874051b0937749211ab0857a18e2b50bb8fefb17ff" exitCode=0 Sep 11 20:26:45.996758 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:45.996639 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ea327d-6f44c946c8-ccphp" Sep 11 20:26:45.996758 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:45.996719 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ea327d-6f44c946c8-ccphp" event={"ID":"b7301644-7f5a-4fb0-a7f3-908f028c938c","Type":"ContainerDied","Data":"a521b6048cebb7aba1b156874051b0937749211ab0857a18e2b50bb8fefb17ff"} Sep 11 20:26:45.996877 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:45.996760 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ea327d-6f44c946c8-ccphp" event={"ID":"b7301644-7f5a-4fb0-a7f3-908f028c938c","Type":"ContainerDied","Data":"a7bd9cd007976cce8c3e61bb2e45ad0fe96bf49c132086cb10782fcdf0839946"} Sep 11 20:26:46.011638 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:46.011283 2546 scope.go:117] "RemoveContainer" containerID="35a82174331b01a8f1c06cade1411877cf5426a6cfd8967901ff8e38f39b29ee" Sep 11 20:26:46.022467 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:46.022446 2546 scope.go:117] "RemoveContainer" containerID="a521b6048cebb7aba1b156874051b0937749211ab0857a18e2b50bb8fefb17ff" Sep 11 20:26:46.031779 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:46.031755 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-ea327d-7999bdbc7-xpxc5"] Sep 11 20:26:46.032433 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:46.032415 2546 scope.go:117] "RemoveContainer" containerID="a521b6048cebb7aba1b156874051b0937749211ab0857a18e2b50bb8fefb17ff" Sep 11 20:26:46.032746 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:26:46.032723 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a521b6048cebb7aba1b156874051b0937749211ab0857a18e2b50bb8fefb17ff\": container with ID starting with a521b6048cebb7aba1b156874051b0937749211ab0857a18e2b50bb8fefb17ff not found: ID does not exist" containerID="a521b6048cebb7aba1b156874051b0937749211ab0857a18e2b50bb8fefb17ff" Sep 11 20:26:46.032819 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:46.032756 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a521b6048cebb7aba1b156874051b0937749211ab0857a18e2b50bb8fefb17ff"} err="failed to get container status \"a521b6048cebb7aba1b156874051b0937749211ab0857a18e2b50bb8fefb17ff\": rpc error: code = NotFound desc = could not find container \"a521b6048cebb7aba1b156874051b0937749211ab0857a18e2b50bb8fefb17ff\": container with ID starting with a521b6048cebb7aba1b156874051b0937749211ab0857a18e2b50bb8fefb17ff not found: ID does not exist" Sep 11 20:26:46.034075 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:46.034054 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-ea327d-7999bdbc7-xpxc5"] Sep 11 20:26:46.048364 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:46.048340 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ea327d-6f44c946c8-ccphp"] Sep 11 20:26:46.059027 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:46.059007 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ea327d-6f44c946c8-ccphp"] Sep 11 20:26:46.072160 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:46.072138 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ea327d-7c65697774-l6rzd"] Sep 11 20:26:46.076371 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:46.076318 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ea327d-7c65697774-l6rzd"] Sep 11 20:26:46.334636 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:46.334526 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:46.451036 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:46.451000 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3abcbcd7-5ecd-417c-a2c2-378966ad14fd" path="/var/lib/kubelet/pods/3abcbcd7-5ecd-417c-a2c2-378966ad14fd/volumes" Sep 11 20:26:46.451369 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:46.451355 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7301644-7f5a-4fb0-a7f3-908f028c938c" path="/var/lib/kubelet/pods/b7301644-7f5a-4fb0-a7f3-908f028c938c/volumes" Sep 11 20:26:46.451740 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:46.451725 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc977a98-1ebc-44cf-8eb0-6f6693b092d3" path="/var/lib/kubelet/pods/dc977a98-1ebc-44cf-8eb0-6f6693b092d3/volumes" Sep 11 20:26:47.467558 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:47.467514 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:48.444875 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:48.444847 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:51.948261 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:51.948225 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:52.045782 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:52.045753 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:52.162599 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:52.162538 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:52.258486 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:52.258406 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:52.697110 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:52.697071 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:53.151497 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:53.151466 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:53.900712 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:53.900670 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:54.757836 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:54.757805 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:55.624139 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:55.624082 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:26:56.407467 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:26:56.407435 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:27:02.116841 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:27:02.116805 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:27:03.006401 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:27:03.006373 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c7e744-855cc7f695-nbv9z"] Sep 11 20:27:03.006658 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:27:03.006631 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c7e744-855cc7f695-nbv9z" podUID="e9c72ca7-dd23-4606-b917-7c7df7de2818" containerName="payload-processing" containerID="cri-o://d0c66203ca3e5a725f09fac8380879ba3a4aa25e3256bd6d3d5235506e321163" gracePeriod=30 Sep 11 20:27:03.021307 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:27:03.021282 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c7e744-6958dc9df4-cd6b2"] Sep 11 20:27:03.021517 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:27:03.021480 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c7e744-6958dc9df4-cd6b2" podUID="74519b19-232e-4c39-905f-d43ab2f80a92" containerName="payload-pre-processing" containerID="cri-o://03158ba7e2fce6ceb7d419bec08f9f98f4a1b42cba13c6229103e47c7c8d8221" gracePeriod=30 Sep 11 20:27:03.061974 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:27:03.061943 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:27:03.163145 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:27:03.163104 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:27:03.243519 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:27:03.243485 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:27:03.299694 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:27:03.299672 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c7e744-855cc7f695-nbv9z" Sep 11 20:27:03.302676 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:27:03.302656 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c7e744-6958dc9df4-cd6b2" Sep 11 20:27:03.395604 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:27:03.395562 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9c72ca7-dd23-4606-b917-7c7df7de2818-plugins-config-volume\") pod \"e9c72ca7-dd23-4606-b917-7c7df7de2818\" (UID: \"e9c72ca7-dd23-4606-b917-7c7df7de2818\") " Sep 11 20:27:03.395808 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:27:03.395626 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74519b19-232e-4c39-905f-d43ab2f80a92-plugins-config-volume\") pod \"74519b19-232e-4c39-905f-d43ab2f80a92\" (UID: \"74519b19-232e-4c39-905f-d43ab2f80a92\") " Sep 11 20:27:03.395808 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:27:03.395684 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z5fnx\" (UniqueName: \"kubernetes.io/projected/74519b19-232e-4c39-905f-d43ab2f80a92-kube-api-access-z5fnx\") pod \"74519b19-232e-4c39-905f-d43ab2f80a92\" (UID: \"74519b19-232e-4c39-905f-d43ab2f80a92\") " Sep 11 20:27:03.395808 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:27:03.395715 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nqtwj\" (UniqueName: \"kubernetes.io/projected/e9c72ca7-dd23-4606-b917-7c7df7de2818-kube-api-access-nqtwj\") pod \"e9c72ca7-dd23-4606-b917-7c7df7de2818\" (UID: \"e9c72ca7-dd23-4606-b917-7c7df7de2818\") " Sep 11 20:27:03.395994 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:27:03.395950 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9c72ca7-dd23-4606-b917-7c7df7de2818-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e9c72ca7-dd23-4606-b917-7c7df7de2818" (UID: "e9c72ca7-dd23-4606-b917-7c7df7de2818"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:27:03.396087 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:27:03.396065 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74519b19-232e-4c39-905f-d43ab2f80a92-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "74519b19-232e-4c39-905f-d43ab2f80a92" (UID: "74519b19-232e-4c39-905f-d43ab2f80a92"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:27:03.397803 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:27:03.397777 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9c72ca7-dd23-4606-b917-7c7df7de2818-kube-api-access-nqtwj" (OuterVolumeSpecName: "kube-api-access-nqtwj") pod "e9c72ca7-dd23-4606-b917-7c7df7de2818" (UID: "e9c72ca7-dd23-4606-b917-7c7df7de2818"). InnerVolumeSpecName "kube-api-access-nqtwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:27:03.397936 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:27:03.397803 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74519b19-232e-4c39-905f-d43ab2f80a92-kube-api-access-z5fnx" (OuterVolumeSpecName: "kube-api-access-z5fnx") pod "74519b19-232e-4c39-905f-d43ab2f80a92" (UID: "74519b19-232e-4c39-905f-d43ab2f80a92"). InnerVolumeSpecName "kube-api-access-z5fnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:27:03.497282 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:27:03.497247 2546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9c72ca7-dd23-4606-b917-7c7df7de2818-plugins-config-volume\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:27:03.497282 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:27:03.497275 2546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74519b19-232e-4c39-905f-d43ab2f80a92-plugins-config-volume\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:27:03.497282 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:27:03.497285 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5fnx\" (UniqueName: \"kubernetes.io/projected/74519b19-232e-4c39-905f-d43ab2f80a92-kube-api-access-z5fnx\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:27:03.497522 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:27:03.497296 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqtwj\" (UniqueName: \"kubernetes.io/projected/e9c72ca7-dd23-4606-b917-7c7df7de2818-kube-api-access-nqtwj\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:27:03.626828 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:27:03.626786 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:27:03.720718 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:27:03.720684 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:27:04.096336 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:27:04.096306 2546 generic.go:356] "Generic (PLEG): container finished" podID="e9c72ca7-dd23-4606-b917-7c7df7de2818" containerID="d0c66203ca3e5a725f09fac8380879ba3a4aa25e3256bd6d3d5235506e321163" exitCode=0 Sep 11 20:27:04.096512 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:27:04.096349 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c7e744-855cc7f695-nbv9z" Sep 11 20:27:04.096512 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:27:04.096389 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c7e744-855cc7f695-nbv9z" event={"ID":"e9c72ca7-dd23-4606-b917-7c7df7de2818","Type":"ContainerDied","Data":"d0c66203ca3e5a725f09fac8380879ba3a4aa25e3256bd6d3d5235506e321163"} Sep 11 20:27:04.096512 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:27:04.096441 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c7e744-855cc7f695-nbv9z" event={"ID":"e9c72ca7-dd23-4606-b917-7c7df7de2818","Type":"ContainerDied","Data":"1d0edc5330178883daf56285bda184c8e4a75c64bfcb14b0a34a11fe59346b38"} Sep 11 20:27:04.096512 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:27:04.096465 2546 scope.go:117] "RemoveContainer" containerID="d0c66203ca3e5a725f09fac8380879ba3a4aa25e3256bd6d3d5235506e321163" Sep 11 20:27:04.098027 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:27:04.097919 2546 generic.go:356] "Generic (PLEG): container finished" podID="74519b19-232e-4c39-905f-d43ab2f80a92" containerID="03158ba7e2fce6ceb7d419bec08f9f98f4a1b42cba13c6229103e47c7c8d8221" exitCode=0 Sep 11 20:27:04.098027 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:27:04.097951 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c7e744-6958dc9df4-cd6b2" event={"ID":"74519b19-232e-4c39-905f-d43ab2f80a92","Type":"ContainerDied","Data":"03158ba7e2fce6ceb7d419bec08f9f98f4a1b42cba13c6229103e47c7c8d8221"} Sep 11 20:27:04.098027 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:27:04.097965 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c7e744-6958dc9df4-cd6b2" Sep 11 20:27:04.098027 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:27:04.098015 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c7e744-6958dc9df4-cd6b2" event={"ID":"74519b19-232e-4c39-905f-d43ab2f80a92","Type":"ContainerDied","Data":"d266d8ddeebd3bcf2fbce3850298b02cef759cbc1aac6756eb3b9e2b7f1b60a0"} Sep 11 20:27:04.107014 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:27:04.106995 2546 scope.go:117] "RemoveContainer" containerID="d0c66203ca3e5a725f09fac8380879ba3a4aa25e3256bd6d3d5235506e321163" Sep 11 20:27:04.107328 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:27:04.107302 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0c66203ca3e5a725f09fac8380879ba3a4aa25e3256bd6d3d5235506e321163\": container with ID starting with d0c66203ca3e5a725f09fac8380879ba3a4aa25e3256bd6d3d5235506e321163 not found: ID does not exist" containerID="d0c66203ca3e5a725f09fac8380879ba3a4aa25e3256bd6d3d5235506e321163" Sep 11 20:27:04.107439 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:27:04.107337 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0c66203ca3e5a725f09fac8380879ba3a4aa25e3256bd6d3d5235506e321163"} err="failed to get container status \"d0c66203ca3e5a725f09fac8380879ba3a4aa25e3256bd6d3d5235506e321163\": rpc error: code = NotFound desc = could not find container \"d0c66203ca3e5a725f09fac8380879ba3a4aa25e3256bd6d3d5235506e321163\": container with ID starting with d0c66203ca3e5a725f09fac8380879ba3a4aa25e3256bd6d3d5235506e321163 not found: ID does not exist" Sep 11 20:27:04.107439 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:27:04.107362 2546 scope.go:117] "RemoveContainer" containerID="03158ba7e2fce6ceb7d419bec08f9f98f4a1b42cba13c6229103e47c7c8d8221" Sep 11 20:27:04.115524 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:27:04.115507 2546 scope.go:117] "RemoveContainer" containerID="03158ba7e2fce6ceb7d419bec08f9f98f4a1b42cba13c6229103e47c7c8d8221" Sep 11 20:27:04.115815 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:27:04.115792 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03158ba7e2fce6ceb7d419bec08f9f98f4a1b42cba13c6229103e47c7c8d8221\": container with ID starting with 03158ba7e2fce6ceb7d419bec08f9f98f4a1b42cba13c6229103e47c7c8d8221 not found: ID does not exist" containerID="03158ba7e2fce6ceb7d419bec08f9f98f4a1b42cba13c6229103e47c7c8d8221" Sep 11 20:27:04.115906 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:27:04.115820 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03158ba7e2fce6ceb7d419bec08f9f98f4a1b42cba13c6229103e47c7c8d8221"} err="failed to get container status \"03158ba7e2fce6ceb7d419bec08f9f98f4a1b42cba13c6229103e47c7c8d8221\": rpc error: code = NotFound desc = could not find container \"03158ba7e2fce6ceb7d419bec08f9f98f4a1b42cba13c6229103e47c7c8d8221\": container with ID starting with 03158ba7e2fce6ceb7d419bec08f9f98f4a1b42cba13c6229103e47c7c8d8221 not found: ID does not exist" Sep 11 20:27:04.125058 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:27:04.125036 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c7e744-855cc7f695-nbv9z"] Sep 11 20:27:04.127181 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:27:04.127160 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c7e744-855cc7f695-nbv9z"] Sep 11 20:27:04.138907 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:27:04.138888 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c7e744-6958dc9df4-cd6b2"] Sep 11 20:27:04.144840 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:27:04.144820 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c7e744-6958dc9df4-cd6b2"] Sep 11 20:27:04.451043 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:27:04.450968 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74519b19-232e-4c39-905f-d43ab2f80a92" path="/var/lib/kubelet/pods/74519b19-232e-4c39-905f-d43ab2f80a92/volumes" Sep 11 20:27:04.451409 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:27:04.451314 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9c72ca7-dd23-4606-b917-7c7df7de2818" path="/var/lib/kubelet/pods/e9c72ca7-dd23-4606-b917-7c7df7de2818/volumes" Sep 11 20:27:04.566990 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:27:04.566955 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:27:05.641052 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:27:05.641019 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:27:09.060476 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:27:09.060445 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:27:10.207222 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:27:10.207186 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:27:10.298548 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:27:10.298519 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:27:10.519773 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:27:10.519694 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:27:34.139312 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:27:34.139275 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:28:14.272781 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:28:14.272746 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:28:15.240517 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:28:15.240486 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:28:18.897621 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:28:18.897565 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:28:49.348671 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:28:49.348640 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:29:08.712043 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:08.711959 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:29:33.995444 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:33.995414 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:29:34.946621 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:34.946559 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:29:35.358994 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:35.358963 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:29:35.656432 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:35.656349 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:29:36.146329 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:36.146281 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:29:37.562902 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:37.562870 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:29:38.293883 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:38.293854 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:29:38.378198 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:38.378165 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:29:38.655762 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:38.655729 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:29:39.481111 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:39.481079 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:29:40.539871 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:40.539842 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-248ab6-8695cbc44c-qgrcz"] Sep 11 20:29:40.540388 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:40.540360 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7301644-7f5a-4fb0-a7f3-908f028c938c" containerName="payload-pre-processing" Sep 11 20:29:40.540388 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:40.540380 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7301644-7f5a-4fb0-a7f3-908f028c938c" containerName="payload-pre-processing" Sep 11 20:29:40.540536 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:40.540406 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2647260f-49fc-450a-9e4b-fda58dfffddc" containerName="maas-api" Sep 11 20:29:40.540536 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:40.540417 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="2647260f-49fc-450a-9e4b-fda58dfffddc" containerName="maas-api" Sep 11 20:29:40.540536 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:40.540428 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3abcbcd7-5ecd-417c-a2c2-378966ad14fd" containerName="payload-processing" Sep 11 20:29:40.540536 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:40.540436 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="3abcbcd7-5ecd-417c-a2c2-378966ad14fd" containerName="payload-processing" Sep 11 20:29:40.540536 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:40.540450 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ad46135-0cc6-41f1-8e00-6a6f58f2be20" containerName="payload-pre-processing" Sep 11 20:29:40.540536 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:40.540457 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ad46135-0cc6-41f1-8e00-6a6f58f2be20" containerName="payload-pre-processing" Sep 11 20:29:40.540536 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:40.540468 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9c72ca7-dd23-4606-b917-7c7df7de2818" containerName="payload-processing" Sep 11 20:29:40.540536 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:40.540476 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9c72ca7-dd23-4606-b917-7c7df7de2818" containerName="payload-processing" Sep 11 20:29:40.540536 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:40.540485 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74519b19-232e-4c39-905f-d43ab2f80a92" containerName="payload-pre-processing" Sep 11 20:29:40.540536 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:40.540490 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="74519b19-232e-4c39-905f-d43ab2f80a92" containerName="payload-pre-processing" Sep 11 20:29:40.540536 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:40.540497 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db0b2703-500b-47e2-9165-8808feae0778" containerName="maas-api" Sep 11 20:29:40.540536 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:40.540502 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="db0b2703-500b-47e2-9165-8808feae0778" containerName="maas-api" Sep 11 20:29:40.540536 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:40.540508 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026" containerName="maas-api" Sep 11 20:29:40.540536 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:40.540516 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026" containerName="maas-api" Sep 11 20:29:40.540536 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:40.540536 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc977a98-1ebc-44cf-8eb0-6f6693b092d3" containerName="maas-api" Sep 11 20:29:40.540536 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:40.540542 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc977a98-1ebc-44cf-8eb0-6f6693b092d3" containerName="maas-api" Sep 11 20:29:40.541273 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:40.540647 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9efbdb9-9b8b-48b1-a3ec-1e85c5eab026" containerName="maas-api" Sep 11 20:29:40.541273 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:40.540658 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7301644-7f5a-4fb0-a7f3-908f028c938c" containerName="payload-pre-processing" Sep 11 20:29:40.541273 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:40.540666 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9c72ca7-dd23-4606-b917-7c7df7de2818" containerName="payload-processing" Sep 11 20:29:40.541273 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:40.540676 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ad46135-0cc6-41f1-8e00-6a6f58f2be20" containerName="payload-pre-processing" Sep 11 20:29:40.541273 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:40.540683 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc977a98-1ebc-44cf-8eb0-6f6693b092d3" containerName="maas-api" Sep 11 20:29:40.541273 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:40.540692 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="db0b2703-500b-47e2-9165-8808feae0778" containerName="maas-api" Sep 11 20:29:40.541273 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:40.540705 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="2647260f-49fc-450a-9e4b-fda58dfffddc" containerName="maas-api" Sep 11 20:29:40.541273 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:40.540713 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="74519b19-232e-4c39-905f-d43ab2f80a92" containerName="payload-pre-processing" Sep 11 20:29:40.541273 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:40.540723 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="3abcbcd7-5ecd-417c-a2c2-378966ad14fd" containerName="payload-processing" Sep 11 20:29:40.544361 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:40.544340 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-248ab6-8695cbc44c-qgrcz" Sep 11 20:29:40.548254 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:40.548236 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-248ab6-serving-cert\"" Sep 11 20:29:40.560981 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:40.560961 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-248ab6-8695cbc44c-qgrcz"] Sep 11 20:29:40.588453 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:40.588431 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:29:40.690398 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:40.690356 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:29:40.723482 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:40.723453 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6477778-0da8-45d2-bc2f-222e7f7a9d2f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-248ab6-8695cbc44c-qgrcz\" (UID: \"d6477778-0da8-45d2-bc2f-222e7f7a9d2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-248ab6-8695cbc44c-qgrcz" Sep 11 20:29:40.723640 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:40.723498 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwgw6\" (UniqueName: \"kubernetes.io/projected/d6477778-0da8-45d2-bc2f-222e7f7a9d2f-kube-api-access-nwgw6\") pod \"maas-api-e2e-ipp-a-248ab6-8695cbc44c-qgrcz\" (UID: \"d6477778-0da8-45d2-bc2f-222e7f7a9d2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-248ab6-8695cbc44c-qgrcz" Sep 11 20:29:40.723640 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:40.723536 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6477778-0da8-45d2-bc2f-222e7f7a9d2f-metrics-certs\") pod \"maas-api-e2e-ipp-a-248ab6-8695cbc44c-qgrcz\" (UID: \"d6477778-0da8-45d2-bc2f-222e7f7a9d2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-248ab6-8695cbc44c-qgrcz" Sep 11 20:29:40.723640 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:40.723607 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6477778-0da8-45d2-bc2f-222e7f7a9d2f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-248ab6-8695cbc44c-qgrcz\" (UID: \"d6477778-0da8-45d2-bc2f-222e7f7a9d2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-248ab6-8695cbc44c-qgrcz" Sep 11 20:29:40.798969 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:40.798880 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:29:40.824083 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:40.824060 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6477778-0da8-45d2-bc2f-222e7f7a9d2f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-248ab6-8695cbc44c-qgrcz\" (UID: \"d6477778-0da8-45d2-bc2f-222e7f7a9d2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-248ab6-8695cbc44c-qgrcz" Sep 11 20:29:40.824197 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:40.824155 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6477778-0da8-45d2-bc2f-222e7f7a9d2f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-248ab6-8695cbc44c-qgrcz\" (UID: \"d6477778-0da8-45d2-bc2f-222e7f7a9d2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-248ab6-8695cbc44c-qgrcz" Sep 11 20:29:40.824241 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:40.824200 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwgw6\" (UniqueName: \"kubernetes.io/projected/d6477778-0da8-45d2-bc2f-222e7f7a9d2f-kube-api-access-nwgw6\") pod \"maas-api-e2e-ipp-a-248ab6-8695cbc44c-qgrcz\" (UID: \"d6477778-0da8-45d2-bc2f-222e7f7a9d2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-248ab6-8695cbc44c-qgrcz" Sep 11 20:29:40.824278 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:40.824240 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6477778-0da8-45d2-bc2f-222e7f7a9d2f-metrics-certs\") pod \"maas-api-e2e-ipp-a-248ab6-8695cbc44c-qgrcz\" (UID: \"d6477778-0da8-45d2-bc2f-222e7f7a9d2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-248ab6-8695cbc44c-qgrcz" Sep 11 20:29:40.824808 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:40.824785 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6477778-0da8-45d2-bc2f-222e7f7a9d2f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-248ab6-8695cbc44c-qgrcz\" (UID: \"d6477778-0da8-45d2-bc2f-222e7f7a9d2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-248ab6-8695cbc44c-qgrcz" Sep 11 20:29:40.826564 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:40.826541 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6477778-0da8-45d2-bc2f-222e7f7a9d2f-metrics-certs\") pod \"maas-api-e2e-ipp-a-248ab6-8695cbc44c-qgrcz\" (UID: \"d6477778-0da8-45d2-bc2f-222e7f7a9d2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-248ab6-8695cbc44c-qgrcz" Sep 11 20:29:40.826564 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:40.826561 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6477778-0da8-45d2-bc2f-222e7f7a9d2f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-248ab6-8695cbc44c-qgrcz\" (UID: \"d6477778-0da8-45d2-bc2f-222e7f7a9d2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-248ab6-8695cbc44c-qgrcz" Sep 11 20:29:40.835856 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:40.835835 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwgw6\" (UniqueName: \"kubernetes.io/projected/d6477778-0da8-45d2-bc2f-222e7f7a9d2f-kube-api-access-nwgw6\") pod \"maas-api-e2e-ipp-a-248ab6-8695cbc44c-qgrcz\" (UID: \"d6477778-0da8-45d2-bc2f-222e7f7a9d2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-248ab6-8695cbc44c-qgrcz" Sep 11 20:29:40.854413 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:40.854386 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-248ab6-8695cbc44c-qgrcz" Sep 11 20:29:41.142104 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:41.141922 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-248ab6-8695cbc44c-qgrcz"] Sep 11 20:29:41.145133 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:29:41.145102 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6477778_0da8_45d2_bc2f_222e7f7a9d2f.slice/crio-499858ee3ff2c27cdf303f36aa048bcb8e1fd91a757aedc65b02f6e6a1db7913 WatchSource:0}: Error finding container 499858ee3ff2c27cdf303f36aa048bcb8e1fd91a757aedc65b02f6e6a1db7913: Status 404 returned error can't find the container with id 499858ee3ff2c27cdf303f36aa048bcb8e1fd91a757aedc65b02f6e6a1db7913 Sep 11 20:29:41.265543 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:41.265512 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:29:41.357799 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:41.357755 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:29:41.485845 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:41.485761 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-248ab6-6dcbcf4bc8-qsqxb"] Sep 11 20:29:41.490710 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:41.490688 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-248ab6-6dcbcf4bc8-qsqxb" Sep 11 20:29:41.493403 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:41.493381 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-248ab6-dockercfg-nvjrg\"" Sep 11 20:29:41.493509 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:41.493381 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-248ab6\"" Sep 11 20:29:41.502403 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:41.502371 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-248ab6-6dcbcf4bc8-qsqxb"] Sep 11 20:29:41.531024 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:41.530981 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3794824d-d5d9-4f3a-bcda-75e80942a10a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-248ab6-6dcbcf4bc8-qsqxb\" (UID: \"3794824d-d5d9-4f3a-bcda-75e80942a10a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-248ab6-6dcbcf4bc8-qsqxb" Sep 11 20:29:41.531187 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:41.531051 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmpl2\" (UniqueName: \"kubernetes.io/projected/3794824d-d5d9-4f3a-bcda-75e80942a10a-kube-api-access-vmpl2\") pod \"payload-pre-processing-e2e-ipp-a-248ab6-6dcbcf4bc8-qsqxb\" (UID: \"3794824d-d5d9-4f3a-bcda-75e80942a10a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-248ab6-6dcbcf4bc8-qsqxb" Sep 11 20:29:41.631853 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:41.631815 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3794824d-d5d9-4f3a-bcda-75e80942a10a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-248ab6-6dcbcf4bc8-qsqxb\" (UID: \"3794824d-d5d9-4f3a-bcda-75e80942a10a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-248ab6-6dcbcf4bc8-qsqxb" Sep 11 20:29:41.632249 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:41.631897 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmpl2\" (UniqueName: \"kubernetes.io/projected/3794824d-d5d9-4f3a-bcda-75e80942a10a-kube-api-access-vmpl2\") pod \"payload-pre-processing-e2e-ipp-a-248ab6-6dcbcf4bc8-qsqxb\" (UID: \"3794824d-d5d9-4f3a-bcda-75e80942a10a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-248ab6-6dcbcf4bc8-qsqxb" Sep 11 20:29:41.632704 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:41.632679 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3794824d-d5d9-4f3a-bcda-75e80942a10a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-248ab6-6dcbcf4bc8-qsqxb\" (UID: \"3794824d-d5d9-4f3a-bcda-75e80942a10a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-248ab6-6dcbcf4bc8-qsqxb" Sep 11 20:29:41.649967 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:41.649937 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmpl2\" (UniqueName: \"kubernetes.io/projected/3794824d-d5d9-4f3a-bcda-75e80942a10a-kube-api-access-vmpl2\") pod \"payload-pre-processing-e2e-ipp-a-248ab6-6dcbcf4bc8-qsqxb\" (UID: \"3794824d-d5d9-4f3a-bcda-75e80942a10a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-248ab6-6dcbcf4bc8-qsqxb" Sep 11 20:29:41.803264 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:41.803188 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-248ab6-6dcbcf4bc8-qsqxb" Sep 11 20:29:41.937751 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:41.937714 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-248ab6-8695cbc44c-qgrcz" event={"ID":"d6477778-0da8-45d2-bc2f-222e7f7a9d2f","Type":"ContainerStarted","Data":"dc95c584fc3ba3bd97ce60881c00fc7b7fd30021b0ed08e0db446ec6fdcef23a"} Sep 11 20:29:41.937948 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:41.937804 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-248ab6-8695cbc44c-qgrcz" event={"ID":"d6477778-0da8-45d2-bc2f-222e7f7a9d2f","Type":"ContainerStarted","Data":"499858ee3ff2c27cdf303f36aa048bcb8e1fd91a757aedc65b02f6e6a1db7913"} Sep 11 20:29:41.937948 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:41.937827 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-248ab6-8695cbc44c-qgrcz" Sep 11 20:29:41.959737 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:41.959694 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-248ab6-8695cbc44c-qgrcz" podStartSLOduration=1.589315 podStartE2EDuration="1.959679669s" totalImagesPullingTime="370.364669ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:29:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:29:41.958884119 +0000 UTC m=+1752.006934463" watchObservedRunningTime="2026-09-11 20:29:41.959679669 +0000 UTC m=+1752.007730013" Sep 11 20:29:42.075886 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:29:42.075853 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3794824d_d5d9_4f3a_bcda_75e80942a10a.slice/crio-33c81ea93c5ade78463abca6e7d0e88f288862682150d8e3f26e3abe0b97a263 WatchSource:0}: Error finding container 33c81ea93c5ade78463abca6e7d0e88f288862682150d8e3f26e3abe0b97a263: Status 404 returned error can't find the container with id 33c81ea93c5ade78463abca6e7d0e88f288862682150d8e3f26e3abe0b97a263 Sep 11 20:29:42.076285 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:42.076261 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-248ab6-6dcbcf4bc8-qsqxb"] Sep 11 20:29:42.944922 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:42.944886 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-248ab6-6dcbcf4bc8-qsqxb" event={"ID":"3794824d-d5d9-4f3a-bcda-75e80942a10a","Type":"ContainerStarted","Data":"5b26e711eba3d21b94bcbeb4f34f7e61f7dc4172e3e2a553800ee7374b1dc866"} Sep 11 20:29:42.945367 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:42.944942 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-248ab6-6dcbcf4bc8-qsqxb" event={"ID":"3794824d-d5d9-4f3a-bcda-75e80942a10a","Type":"ContainerStarted","Data":"33c81ea93c5ade78463abca6e7d0e88f288862682150d8e3f26e3abe0b97a263"} Sep 11 20:29:42.945367 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:42.945171 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-248ab6-6dcbcf4bc8-qsqxb" Sep 11 20:29:42.967176 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:42.967127 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-248ab6-6dcbcf4bc8-qsqxb" podStartSLOduration=1.967112636 podStartE2EDuration="1.967112636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:29:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:29:42.964433726 +0000 UTC m=+1753.012484093" watchObservedRunningTime="2026-09-11 20:29:42.967112636 +0000 UTC m=+1753.015162981" Sep 11 20:29:47.950458 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:47.950432 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-248ab6-8695cbc44c-qgrcz" Sep 11 20:29:49.354200 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:49.354165 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:29:53.955671 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:29:53.955627 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-248ab6-6dcbcf4bc8-qsqxb" Sep 11 20:30:04.198933 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:30:04.198901 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:30:05.116409 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:30:05.116373 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:30:06.002370 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:30:06.002310 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:30:06.634716 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:30:06.634684 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:30:08.373892 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:30:08.373859 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:30:09.049434 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:30:09.049399 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:30:09.367496 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:30:09.367430 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:30:09.705589 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:30:09.705505 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:30:11.728803 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:30:11.728770 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:30:12.447262 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:30:12.447215 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:30:12.539374 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:30:12.539343 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:30:12.742324 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:30:12.742242 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:30:13.778508 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:30:13.778475 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:30:14.556470 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:30:14.556436 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:30:15.386330 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:30:15.386301 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:30:15.486750 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:30:15.486720 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:30:15.730809 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:30:15.730731 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:30:15.890811 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:30:15.890777 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:30:15.990091 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:30:15.990013 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:30:16.073331 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:30:16.073301 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b6453d-66bf6d488b-cvxbp"] Sep 11 20:30:16.077085 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:30:16.077067 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6453d-66bf6d488b-cvxbp" Sep 11 20:30:16.079720 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:30:16.079693 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b6453d-dockercfg-pp26b\"" Sep 11 20:30:16.079859 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:30:16.079693 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b6453d\"" Sep 11 20:30:16.086924 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:30:16.086903 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b6453d-66bf6d488b-cvxbp"] Sep 11 20:30:16.256783 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:30:16.256704 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cc670c4-aa51-441e-9027-68c0886aab39-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b6453d-66bf6d488b-cvxbp\" (UID: \"6cc670c4-aa51-441e-9027-68c0886aab39\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6453d-66bf6d488b-cvxbp" Sep 11 20:30:16.256927 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:30:16.256788 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65l5n\" (UniqueName: \"kubernetes.io/projected/6cc670c4-aa51-441e-9027-68c0886aab39-kube-api-access-65l5n\") pod \"payload-pre-processing-e2e-ipp-b-b6453d-66bf6d488b-cvxbp\" (UID: \"6cc670c4-aa51-441e-9027-68c0886aab39\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6453d-66bf6d488b-cvxbp" Sep 11 20:30:16.357625 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:30:16.357571 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65l5n\" (UniqueName: \"kubernetes.io/projected/6cc670c4-aa51-441e-9027-68c0886aab39-kube-api-access-65l5n\") pod \"payload-pre-processing-e2e-ipp-b-b6453d-66bf6d488b-cvxbp\" (UID: \"6cc670c4-aa51-441e-9027-68c0886aab39\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6453d-66bf6d488b-cvxbp" Sep 11 20:30:16.357787 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:30:16.357662 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cc670c4-aa51-441e-9027-68c0886aab39-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b6453d-66bf6d488b-cvxbp\" (UID: \"6cc670c4-aa51-441e-9027-68c0886aab39\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6453d-66bf6d488b-cvxbp" Sep 11 20:30:16.358359 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:30:16.358335 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cc670c4-aa51-441e-9027-68c0886aab39-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b6453d-66bf6d488b-cvxbp\" (UID: \"6cc670c4-aa51-441e-9027-68c0886aab39\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6453d-66bf6d488b-cvxbp" Sep 11 20:30:16.365401 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:30:16.365379 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65l5n\" (UniqueName: \"kubernetes.io/projected/6cc670c4-aa51-441e-9027-68c0886aab39-kube-api-access-65l5n\") pod \"payload-pre-processing-e2e-ipp-b-b6453d-66bf6d488b-cvxbp\" (UID: \"6cc670c4-aa51-441e-9027-68c0886aab39\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6453d-66bf6d488b-cvxbp" Sep 11 20:30:16.388888 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:30:16.388864 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6453d-66bf6d488b-cvxbp" Sep 11 20:30:16.668145 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:30:16.668121 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b6453d-66bf6d488b-cvxbp"] Sep 11 20:30:16.670069 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:30:16.670034 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cc670c4_aa51_441e_9027_68c0886aab39.slice/crio-ee80475fa8869b0173fb5700fb4c4578ef8311f382e2137c0065bef938d2b91f WatchSource:0}: Error finding container ee80475fa8869b0173fb5700fb4c4578ef8311f382e2137c0065bef938d2b91f: Status 404 returned error can't find the container with id ee80475fa8869b0173fb5700fb4c4578ef8311f382e2137c0065bef938d2b91f Sep 11 20:30:17.134346 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:30:17.134308 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6453d-66bf6d488b-cvxbp" event={"ID":"6cc670c4-aa51-441e-9027-68c0886aab39","Type":"ContainerStarted","Data":"19dd28fe4ed5ed359be2054337793baebf412af8d336bb590a272317db95858b"} Sep 11 20:30:17.134602 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:30:17.134378 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6453d-66bf6d488b-cvxbp" event={"ID":"6cc670c4-aa51-441e-9027-68c0886aab39","Type":"ContainerStarted","Data":"ee80475fa8869b0173fb5700fb4c4578ef8311f382e2137c0065bef938d2b91f"} Sep 11 20:30:17.134602 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:30:17.134449 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6453d-66bf6d488b-cvxbp" Sep 11 20:30:17.153105 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:30:17.153054 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6453d-66bf6d488b-cvxbp" podStartSLOduration=1.153035576 podStartE2EDuration="1.153035576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:30:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:30:17.151918852 +0000 UTC m=+1787.199969198" watchObservedRunningTime="2026-09-11 20:30:17.153035576 +0000 UTC m=+1787.201085923" Sep 11 20:30:28.143135 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:30:28.143049 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6453d-66bf6d488b-cvxbp" Sep 11 20:30:37.462811 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:30:37.462773 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:30:37.618798 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:30:37.618766 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:30:38.539503 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:30:38.539464 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:30:39.431568 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:30:39.431531 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:30:40.883300 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:30:40.883266 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:30:41.042262 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:30:41.042225 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:31:00.602197 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:00.602168 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:31:21.630788 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:21.630752 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:31:25.190300 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:25.190269 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:31:34.145026 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:34.144995 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:31:34.413864 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:34.413781 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x"] Sep 11 20:31:34.417463 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:34.417441 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x" Sep 11 20:31:34.419715 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:34.419695 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-5de5ea-openshift-default-dockercfg-fgnd5\"" Sep 11 20:31:34.421748 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:34.421718 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0675d2f-779a-4e4d-8669-fa774841dab4-istio-data\") pod \"e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x\" (UID: \"f0675d2f-779a-4e4d-8669-fa774841dab4\") " pod="openshift-ingress/e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x" Sep 11 20:31:34.421843 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:34.421748 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0675d2f-779a-4e4d-8669-fa774841dab4-istiod-ca-cert\") pod \"e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x\" (UID: \"f0675d2f-779a-4e4d-8669-fa774841dab4\") " pod="openshift-ingress/e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x" Sep 11 20:31:34.421843 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:34.421789 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0675d2f-779a-4e4d-8669-fa774841dab4-credential-socket\") pod \"e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x\" (UID: \"f0675d2f-779a-4e4d-8669-fa774841dab4\") " pod="openshift-ingress/e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x" Sep 11 20:31:34.421949 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:34.421833 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0675d2f-779a-4e4d-8669-fa774841dab4-workload-socket\") pod \"e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x\" (UID: \"f0675d2f-779a-4e4d-8669-fa774841dab4\") " pod="openshift-ingress/e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x" Sep 11 20:31:34.421949 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:34.421895 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0675d2f-779a-4e4d-8669-fa774841dab4-workload-certs\") pod \"e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x\" (UID: \"f0675d2f-779a-4e4d-8669-fa774841dab4\") " pod="openshift-ingress/e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x" Sep 11 20:31:34.422030 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:34.421980 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0675d2f-779a-4e4d-8669-fa774841dab4-istio-podinfo\") pod \"e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x\" (UID: \"f0675d2f-779a-4e4d-8669-fa774841dab4\") " pod="openshift-ingress/e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x" Sep 11 20:31:34.422030 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:34.422004 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0675d2f-779a-4e4d-8669-fa774841dab4-istio-envoy\") pod \"e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x\" (UID: \"f0675d2f-779a-4e4d-8669-fa774841dab4\") " pod="openshift-ingress/e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x" Sep 11 20:31:34.422112 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:34.422028 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdnpk\" (UniqueName: \"kubernetes.io/projected/f0675d2f-779a-4e4d-8669-fa774841dab4-kube-api-access-sdnpk\") pod \"e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x\" (UID: \"f0675d2f-779a-4e4d-8669-fa774841dab4\") " pod="openshift-ingress/e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x" Sep 11 20:31:34.422112 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:34.422054 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0675d2f-779a-4e4d-8669-fa774841dab4-istio-token\") pod \"e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x\" (UID: \"f0675d2f-779a-4e4d-8669-fa774841dab4\") " pod="openshift-ingress/e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x" Sep 11 20:31:34.429609 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:34.429562 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x"] Sep 11 20:31:34.439910 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:34.439887 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:31:34.522527 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:34.522496 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0675d2f-779a-4e4d-8669-fa774841dab4-istio-data\") pod \"e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x\" (UID: \"f0675d2f-779a-4e4d-8669-fa774841dab4\") " pod="openshift-ingress/e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x" Sep 11 20:31:34.522527 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:34.522531 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0675d2f-779a-4e4d-8669-fa774841dab4-istiod-ca-cert\") pod \"e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x\" (UID: \"f0675d2f-779a-4e4d-8669-fa774841dab4\") " pod="openshift-ingress/e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x" Sep 11 20:31:34.522800 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:34.522737 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0675d2f-779a-4e4d-8669-fa774841dab4-credential-socket\") pod \"e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x\" (UID: \"f0675d2f-779a-4e4d-8669-fa774841dab4\") " pod="openshift-ingress/e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x" Sep 11 20:31:34.522800 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:34.522787 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0675d2f-779a-4e4d-8669-fa774841dab4-workload-socket\") pod \"e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x\" (UID: \"f0675d2f-779a-4e4d-8669-fa774841dab4\") " pod="openshift-ingress/e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x" Sep 11 20:31:34.522927 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:34.522836 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0675d2f-779a-4e4d-8669-fa774841dab4-workload-certs\") pod \"e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x\" (UID: \"f0675d2f-779a-4e4d-8669-fa774841dab4\") " pod="openshift-ingress/e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x" Sep 11 20:31:34.522989 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:34.522979 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0675d2f-779a-4e4d-8669-fa774841dab4-istio-podinfo\") pod \"e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x\" (UID: \"f0675d2f-779a-4e4d-8669-fa774841dab4\") " pod="openshift-ingress/e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x" Sep 11 20:31:34.523051 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:34.523025 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0675d2f-779a-4e4d-8669-fa774841dab4-istio-envoy\") pod \"e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x\" (UID: \"f0675d2f-779a-4e4d-8669-fa774841dab4\") " pod="openshift-ingress/e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x" Sep 11 20:31:34.523106 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:34.523054 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdnpk\" (UniqueName: \"kubernetes.io/projected/f0675d2f-779a-4e4d-8669-fa774841dab4-kube-api-access-sdnpk\") pod \"e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x\" (UID: \"f0675d2f-779a-4e4d-8669-fa774841dab4\") " pod="openshift-ingress/e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x" Sep 11 20:31:34.523161 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:34.523097 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0675d2f-779a-4e4d-8669-fa774841dab4-istio-token\") pod \"e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x\" (UID: \"f0675d2f-779a-4e4d-8669-fa774841dab4\") " pod="openshift-ingress/e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x" Sep 11 20:31:34.523161 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:34.523145 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0675d2f-779a-4e4d-8669-fa774841dab4-credential-socket\") pod \"e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x\" (UID: \"f0675d2f-779a-4e4d-8669-fa774841dab4\") " pod="openshift-ingress/e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x" Sep 11 20:31:34.523263 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:34.523187 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0675d2f-779a-4e4d-8669-fa774841dab4-workload-socket\") pod \"e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x\" (UID: \"f0675d2f-779a-4e4d-8669-fa774841dab4\") " pod="openshift-ingress/e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x" Sep 11 20:31:34.523478 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:34.523454 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0675d2f-779a-4e4d-8669-fa774841dab4-istiod-ca-cert\") pod \"e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x\" (UID: \"f0675d2f-779a-4e4d-8669-fa774841dab4\") " pod="openshift-ingress/e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x" Sep 11 20:31:34.523746 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:34.523722 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0675d2f-779a-4e4d-8669-fa774841dab4-istio-data\") pod \"e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x\" (UID: \"f0675d2f-779a-4e4d-8669-fa774841dab4\") " pod="openshift-ingress/e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x" Sep 11 20:31:34.523851 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:34.523749 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0675d2f-779a-4e4d-8669-fa774841dab4-workload-certs\") pod \"e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x\" (UID: \"f0675d2f-779a-4e4d-8669-fa774841dab4\") " pod="openshift-ingress/e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x" Sep 11 20:31:34.526089 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:34.526065 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0675d2f-779a-4e4d-8669-fa774841dab4-istio-envoy\") pod \"e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x\" (UID: \"f0675d2f-779a-4e4d-8669-fa774841dab4\") " pod="openshift-ingress/e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x" Sep 11 20:31:34.526378 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:34.526329 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0675d2f-779a-4e4d-8669-fa774841dab4-istio-podinfo\") pod \"e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x\" (UID: \"f0675d2f-779a-4e4d-8669-fa774841dab4\") " pod="openshift-ingress/e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x" Sep 11 20:31:34.531091 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:34.531065 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0675d2f-779a-4e4d-8669-fa774841dab4-istio-token\") pod \"e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x\" (UID: \"f0675d2f-779a-4e4d-8669-fa774841dab4\") " pod="openshift-ingress/e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x" Sep 11 20:31:34.531382 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:34.531356 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdnpk\" (UniqueName: \"kubernetes.io/projected/f0675d2f-779a-4e4d-8669-fa774841dab4-kube-api-access-sdnpk\") pod \"e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x\" (UID: \"f0675d2f-779a-4e4d-8669-fa774841dab4\") " pod="openshift-ingress/e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x" Sep 11 20:31:34.730037 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:34.729958 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x" Sep 11 20:31:34.903189 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:34.903156 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:31:35.005029 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:35.005001 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x"] Sep 11 20:31:35.006604 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:31:35.006566 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0675d2f_779a_4e4d_8669_fa774841dab4.slice/crio-4d14d89514b34bfee6033c35589e605b439909c359a21e32509945c1dc8654a1 WatchSource:0}: Error finding container 4d14d89514b34bfee6033c35589e605b439909c359a21e32509945c1dc8654a1: Status 404 returned error can't find the container with id 4d14d89514b34bfee6033c35589e605b439909c359a21e32509945c1dc8654a1 Sep 11 20:31:35.008937 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:35.008906 2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 20:31:35.009007 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:35.008971 2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 20:31:35.009007 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:35.008996 2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 20:31:35.556174 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:35.556134 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x" event={"ID":"f0675d2f-779a-4e4d-8669-fa774841dab4","Type":"ContainerStarted","Data":"eea6ff6be95f0d3b57a81db6cb97d410bf70cd2108448e8764bcac1c927c36cf"} Sep 11 20:31:35.556542 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:35.556215 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x" event={"ID":"f0675d2f-779a-4e4d-8669-fa774841dab4","Type":"ContainerStarted","Data":"4d14d89514b34bfee6033c35589e605b439909c359a21e32509945c1dc8654a1"} Sep 11 20:31:35.576784 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:35.576731 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x" podStartSLOduration=1.576711025 podStartE2EDuration="1.576711025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:31:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:31:35.572450317 +0000 UTC m=+1865.620500661" watchObservedRunningTime="2026-09-11 20:31:35.576711025 +0000 UTC m=+1865.624761370" Sep 11 20:31:35.730166 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:35.730133 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x" Sep 11 20:31:36.735211 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:36.735183 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x" Sep 11 20:31:36.742524 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:36.742502 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:31:37.519504 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:37.519465 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:31:37.566971 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:37.566943 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x" Sep 11 20:31:37.567818 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:37.567799 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x" Sep 11 20:31:37.593530 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:37.593496 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:31:38.520226 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:38.520196 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:31:39.530271 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:39.530236 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:31:40.187760 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:40.187728 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:31:40.932917 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:40.932882 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:31:41.033090 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:41.033038 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:31:41.133789 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:41.133759 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:31:41.395937 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:41.395905 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:31:41.477728 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:41.477690 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:31:41.527515 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:41.527485 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-5de5ea-78fff77597-w67ww"] Sep 11 20:31:41.531306 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:41.531288 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5de5ea-78fff77597-w67ww" Sep 11 20:31:41.533470 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:41.533451 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-5de5ea\"" Sep 11 20:31:41.533613 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:41.533572 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-5de5ea-dockercfg-pp2k8\"" Sep 11 20:31:41.544973 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:41.544950 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-5de5ea-78fff77597-w67ww"] Sep 11 20:31:41.594005 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:41.593975 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmfvz\" (UniqueName: \"kubernetes.io/projected/fdb1c9ca-d879-4d6f-b470-022dae6a45cf-kube-api-access-kmfvz\") pod \"payload-processing-e2e-ipp-cleanup-5de5ea-78fff77597-w67ww\" (UID: \"fdb1c9ca-d879-4d6f-b470-022dae6a45cf\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5de5ea-78fff77597-w67ww" Sep 11 20:31:41.594145 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:41.594039 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdb1c9ca-d879-4d6f-b470-022dae6a45cf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-5de5ea-78fff77597-w67ww\" (UID: \"fdb1c9ca-d879-4d6f-b470-022dae6a45cf\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5de5ea-78fff77597-w67ww" Sep 11 20:31:41.695270 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:41.695172 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmfvz\" (UniqueName: \"kubernetes.io/projected/fdb1c9ca-d879-4d6f-b470-022dae6a45cf-kube-api-access-kmfvz\") pod \"payload-processing-e2e-ipp-cleanup-5de5ea-78fff77597-w67ww\" (UID: \"fdb1c9ca-d879-4d6f-b470-022dae6a45cf\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5de5ea-78fff77597-w67ww" Sep 11 20:31:41.695270 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:41.695252 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdb1c9ca-d879-4d6f-b470-022dae6a45cf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-5de5ea-78fff77597-w67ww\" (UID: \"fdb1c9ca-d879-4d6f-b470-022dae6a45cf\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5de5ea-78fff77597-w67ww" Sep 11 20:31:41.695950 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:41.695931 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdb1c9ca-d879-4d6f-b470-022dae6a45cf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-5de5ea-78fff77597-w67ww\" (UID: \"fdb1c9ca-d879-4d6f-b470-022dae6a45cf\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5de5ea-78fff77597-w67ww" Sep 11 20:31:41.702209 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:41.702186 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmfvz\" (UniqueName: \"kubernetes.io/projected/fdb1c9ca-d879-4d6f-b470-022dae6a45cf-kube-api-access-kmfvz\") pod \"payload-processing-e2e-ipp-cleanup-5de5ea-78fff77597-w67ww\" (UID: \"fdb1c9ca-d879-4d6f-b470-022dae6a45cf\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5de5ea-78fff77597-w67ww" Sep 11 20:31:41.804310 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:41.804280 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:31:41.842966 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:41.842938 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5de5ea-78fff77597-w67ww" Sep 11 20:31:42.117531 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:42.117505 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-5de5ea-78fff77597-w67ww"] Sep 11 20:31:42.119666 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:31:42.119627 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdb1c9ca_d879_4d6f_b470_022dae6a45cf.slice/crio-b552d9dd0bc652d044ee0c6b8abc18c6472f2a99ff253cc2b784e8503e6d8160 WatchSource:0}: Error finding container b552d9dd0bc652d044ee0c6b8abc18c6472f2a99ff253cc2b784e8503e6d8160: Status 404 returned error can't find the container with id b552d9dd0bc652d044ee0c6b8abc18c6472f2a99ff253cc2b784e8503e6d8160 Sep 11 20:31:42.582947 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:42.582840 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:31:42.593254 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:42.593224 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5de5ea-78fff77597-w67ww" event={"ID":"fdb1c9ca-d879-4d6f-b470-022dae6a45cf","Type":"ContainerStarted","Data":"c94c65eb6cec3303157585bcea3060fe8462322c19f420e439b490631d078cf7"} Sep 11 20:31:42.593401 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:42.593284 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5de5ea-78fff77597-w67ww" event={"ID":"fdb1c9ca-d879-4d6f-b470-022dae6a45cf","Type":"ContainerStarted","Data":"b552d9dd0bc652d044ee0c6b8abc18c6472f2a99ff253cc2b784e8503e6d8160"} Sep 11 20:31:42.593485 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:42.593410 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5de5ea-78fff77597-w67ww" Sep 11 20:31:42.610237 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:42.610193 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5de5ea-78fff77597-w67ww" podStartSLOduration=1.6101789659999999 podStartE2EDuration="1.610178966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:31:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:31:42.608169873 +0000 UTC m=+1872.656220245" watchObservedRunningTime="2026-09-11 20:31:42.610178966 +0000 UTC m=+1872.658229310" Sep 11 20:31:53.601312 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:53.601233 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5de5ea-78fff77597-w67ww" Sep 11 20:31:57.686928 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:57.686841 2546 generic.go:356] "Generic (PLEG): container finished" podID="f0675d2f-779a-4e4d-8669-fa774841dab4" containerID="eea6ff6be95f0d3b57a81db6cb97d410bf70cd2108448e8764bcac1c927c36cf" exitCode=0 Sep 11 20:31:57.687330 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:57.686925 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x" event={"ID":"f0675d2f-779a-4e4d-8669-fa774841dab4","Type":"ContainerDied","Data":"eea6ff6be95f0d3b57a81db6cb97d410bf70cd2108448e8764bcac1c927c36cf"} Sep 11 20:31:57.687375 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:57.687362 2546 scope.go:117] "RemoveContainer" containerID="eea6ff6be95f0d3b57a81db6cb97d410bf70cd2108448e8764bcac1c927c36cf" Sep 11 20:31:57.688598 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:57.688555 2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 20:31:57.688652 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:57.688639 2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 20:31:57.688697 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:57.688670 2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 20:31:58.694285 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:58.694250 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x" event={"ID":"f0675d2f-779a-4e4d-8669-fa774841dab4","Type":"ContainerStarted","Data":"49a6942e3a273eae6984711f7a933e7c1caef6df59afc8636cc1997a678bbe8d"} Sep 11 20:31:58.730862 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:58.730836 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x" Sep 11 20:31:58.731013 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:58.730991 2546 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Sep 11 20:31:58.731060 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:58.731041 2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x" podUID="f0675d2f-779a-4e4d-8669-fa774841dab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Sep 11 20:31:59.731124 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:59.731093 2546 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Sep 11 20:31:59.731714 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:31:59.731666 2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x" podUID="f0675d2f-779a-4e4d-8669-fa774841dab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Sep 11 20:32:00.731369 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:00.731331 2546 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Sep 11 20:32:00.731970 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:00.731938 2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x" podUID="f0675d2f-779a-4e4d-8669-fa774841dab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Sep 11 20:32:01.732969 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:01.732929 2546 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Sep 11 20:32:01.735334 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:01.735300 2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x" podUID="f0675d2f-779a-4e4d-8669-fa774841dab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Sep 11 20:32:02.730901 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:02.730859 2546 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Sep 11 20:32:02.731102 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:02.730936 2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x" podUID="f0675d2f-779a-4e4d-8669-fa774841dab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Sep 11 20:32:03.008549 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:03.008448 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:32:03.730807 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:03.730765 2546 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Sep 11 20:32:03.731028 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:03.730866 2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x" podUID="f0675d2f-779a-4e4d-8669-fa774841dab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Sep 11 20:32:03.920184 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:03.920129 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:32:04.610524 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:04.610489 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:32:04.734534 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:04.734505 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x" Sep 11 20:32:04.734819 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:04.734794 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x" Sep 11 20:32:04.735490 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:04.735472 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x" Sep 11 20:32:05.503045 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:05.503009 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:32:06.146909 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:06.146880 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:32:06.786924 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:06.786888 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:32:07.515258 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:07.515225 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:32:08.853761 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:08.853726 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:32:09.715418 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:09.715385 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:32:10.361288 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:10.361262 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:32:11.115889 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:11.115856 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:32:11.741622 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:11.741568 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:32:12.393815 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:12.393783 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:32:19.219924 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:19.219885 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:32:19.245850 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:19.245820 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-5de5ea-78fff77597-w67ww"] Sep 11 20:32:19.246100 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:19.246076 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5de5ea-78fff77597-w67ww" podUID="fdb1c9ca-d879-4d6f-b470-022dae6a45cf" containerName="payload-processing" containerID="cri-o://c94c65eb6cec3303157585bcea3060fe8462322c19f420e439b490631d078cf7" gracePeriod=30 Sep 11 20:32:19.320362 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:19.320331 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:32:19.404725 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:19.404691 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:32:19.463556 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:19.463505 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:32:19.540845 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:19.540823 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5de5ea-78fff77597-w67ww" Sep 11 20:32:19.593869 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:19.593843 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdb1c9ca-d879-4d6f-b470-022dae6a45cf-plugins-config-volume\") pod \"fdb1c9ca-d879-4d6f-b470-022dae6a45cf\" (UID: \"fdb1c9ca-d879-4d6f-b470-022dae6a45cf\") " Sep 11 20:32:19.594016 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:19.593890 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kmfvz\" (UniqueName: \"kubernetes.io/projected/fdb1c9ca-d879-4d6f-b470-022dae6a45cf-kube-api-access-kmfvz\") pod \"fdb1c9ca-d879-4d6f-b470-022dae6a45cf\" (UID: \"fdb1c9ca-d879-4d6f-b470-022dae6a45cf\") " Sep 11 20:32:19.594258 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:19.594231 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdb1c9ca-d879-4d6f-b470-022dae6a45cf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fdb1c9ca-d879-4d6f-b470-022dae6a45cf" (UID: "fdb1c9ca-d879-4d6f-b470-022dae6a45cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:32:19.596009 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:19.595983 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdb1c9ca-d879-4d6f-b470-022dae6a45cf-kube-api-access-kmfvz" (OuterVolumeSpecName: "kube-api-access-kmfvz") pod "fdb1c9ca-d879-4d6f-b470-022dae6a45cf" (UID: "fdb1c9ca-d879-4d6f-b470-022dae6a45cf"). InnerVolumeSpecName "kube-api-access-kmfvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:32:19.695621 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:19.695561 2546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdb1c9ca-d879-4d6f-b470-022dae6a45cf-plugins-config-volume\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:32:19.695621 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:19.695618 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmfvz\" (UniqueName: \"kubernetes.io/projected/fdb1c9ca-d879-4d6f-b470-022dae6a45cf-kube-api-access-kmfvz\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:32:19.758640 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:19.758608 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:32:19.811484 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:19.811394 2546 generic.go:356] "Generic (PLEG): container finished" podID="fdb1c9ca-d879-4d6f-b470-022dae6a45cf" containerID="c94c65eb6cec3303157585bcea3060fe8462322c19f420e439b490631d078cf7" exitCode=0 Sep 11 20:32:19.811484 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:19.811447 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5de5ea-78fff77597-w67ww" Sep 11 20:32:19.811731 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:19.811485 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5de5ea-78fff77597-w67ww" event={"ID":"fdb1c9ca-d879-4d6f-b470-022dae6a45cf","Type":"ContainerDied","Data":"c94c65eb6cec3303157585bcea3060fe8462322c19f420e439b490631d078cf7"} Sep 11 20:32:19.811731 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:19.811537 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5de5ea-78fff77597-w67ww" event={"ID":"fdb1c9ca-d879-4d6f-b470-022dae6a45cf","Type":"ContainerDied","Data":"b552d9dd0bc652d044ee0c6b8abc18c6472f2a99ff253cc2b784e8503e6d8160"} Sep 11 20:32:19.811731 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:19.811559 2546 scope.go:117] "RemoveContainer" containerID="c94c65eb6cec3303157585bcea3060fe8462322c19f420e439b490631d078cf7" Sep 11 20:32:19.821020 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:19.820999 2546 scope.go:117] "RemoveContainer" containerID="c94c65eb6cec3303157585bcea3060fe8462322c19f420e439b490631d078cf7" Sep 11 20:32:19.821287 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:32:19.821266 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c94c65eb6cec3303157585bcea3060fe8462322c19f420e439b490631d078cf7\": container with ID starting with c94c65eb6cec3303157585bcea3060fe8462322c19f420e439b490631d078cf7 not found: ID does not exist" containerID="c94c65eb6cec3303157585bcea3060fe8462322c19f420e439b490631d078cf7" Sep 11 20:32:19.821389 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:19.821294 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c94c65eb6cec3303157585bcea3060fe8462322c19f420e439b490631d078cf7"} err="failed to get container status \"c94c65eb6cec3303157585bcea3060fe8462322c19f420e439b490631d078cf7\": rpc error: code = NotFound desc = could not find container \"c94c65eb6cec3303157585bcea3060fe8462322c19f420e439b490631d078cf7\": container with ID starting with c94c65eb6cec3303157585bcea3060fe8462322c19f420e439b490631d078cf7 not found: ID does not exist" Sep 11 20:32:19.834966 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:19.834946 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-5de5ea-78fff77597-w67ww"] Sep 11 20:32:19.837722 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:19.837695 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:32:19.838713 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:19.838696 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-5de5ea-78fff77597-w67ww"] Sep 11 20:32:20.457437 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:20.457404 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdb1c9ca-d879-4d6f-b470-022dae6a45cf" path="/var/lib/kubelet/pods/fdb1c9ca-d879-4d6f-b470-022dae6a45cf/volumes" Sep 11 20:32:20.498814 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:20.498779 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:32:21.302951 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:21.302908 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:32:21.935485 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:21.935454 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:32:22.581680 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:22.581642 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:32:25.891212 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:25.891181 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:32:25.954256 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:25.954227 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:32:26.113482 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:26.113439 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x"] Sep 11 20:32:26.113705 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:26.113681 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x" podUID="f0675d2f-779a-4e4d-8669-fa774841dab4" containerName="istio-proxy" containerID="cri-o://49a6942e3a273eae6984711f7a933e7c1caef6df59afc8636cc1997a678bbe8d" gracePeriod=30 Sep 11 20:32:26.131627 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:26.131565 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:32:29.895309 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:29.895274 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:32:30.680978 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:30.680934 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:32:31.364102 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:31.364074 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x" Sep 11 20:32:31.428959 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:31.428885 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:32:31.508852 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:31.508820 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0675d2f-779a-4e4d-8669-fa774841dab4-istio-envoy\") pod \"f0675d2f-779a-4e4d-8669-fa774841dab4\" (UID: \"f0675d2f-779a-4e4d-8669-fa774841dab4\") " Sep 11 20:32:31.509029 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:31.508878 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0675d2f-779a-4e4d-8669-fa774841dab4-workload-socket\") pod \"f0675d2f-779a-4e4d-8669-fa774841dab4\" (UID: \"f0675d2f-779a-4e4d-8669-fa774841dab4\") " Sep 11 20:32:31.509029 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:31.508950 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0675d2f-779a-4e4d-8669-fa774841dab4-istio-token\") pod \"f0675d2f-779a-4e4d-8669-fa774841dab4\" (UID: \"f0675d2f-779a-4e4d-8669-fa774841dab4\") " Sep 11 20:32:31.509029 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:31.509009 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0675d2f-779a-4e4d-8669-fa774841dab4-istiod-ca-cert\") pod \"f0675d2f-779a-4e4d-8669-fa774841dab4\" (UID: \"f0675d2f-779a-4e4d-8669-fa774841dab4\") " Sep 11 20:32:31.509204 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:31.509074 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0675d2f-779a-4e4d-8669-fa774841dab4-istio-podinfo\") pod \"f0675d2f-779a-4e4d-8669-fa774841dab4\" (UID: \"f0675d2f-779a-4e4d-8669-fa774841dab4\") " Sep 11 20:32:31.509204 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:31.509107 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0675d2f-779a-4e4d-8669-fa774841dab4-workload-certs\") pod \"f0675d2f-779a-4e4d-8669-fa774841dab4\" (UID: \"f0675d2f-779a-4e4d-8669-fa774841dab4\") " Sep 11 20:32:31.509306 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:31.509222 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0675d2f-779a-4e4d-8669-fa774841dab4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f0675d2f-779a-4e4d-8669-fa774841dab4" (UID: "f0675d2f-779a-4e4d-8669-fa774841dab4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:32:31.509306 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:31.509262 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0675d2f-779a-4e4d-8669-fa774841dab4-credential-socket\") pod \"f0675d2f-779a-4e4d-8669-fa774841dab4\" (UID: \"f0675d2f-779a-4e4d-8669-fa774841dab4\") " Sep 11 20:32:31.509413 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:31.509313 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sdnpk\" (UniqueName: \"kubernetes.io/projected/f0675d2f-779a-4e4d-8669-fa774841dab4-kube-api-access-sdnpk\") pod \"f0675d2f-779a-4e4d-8669-fa774841dab4\" (UID: \"f0675d2f-779a-4e4d-8669-fa774841dab4\") " Sep 11 20:32:31.509413 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:31.509347 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0675d2f-779a-4e4d-8669-fa774841dab4-istio-data\") pod \"f0675d2f-779a-4e4d-8669-fa774841dab4\" (UID: \"f0675d2f-779a-4e4d-8669-fa774841dab4\") " Sep 11 20:32:31.509413 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:31.509352 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0675d2f-779a-4e4d-8669-fa774841dab4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f0675d2f-779a-4e4d-8669-fa774841dab4" (UID: "f0675d2f-779a-4e4d-8669-fa774841dab4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:32:31.509613 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:31.509443 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0675d2f-779a-4e4d-8669-fa774841dab4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f0675d2f-779a-4e4d-8669-fa774841dab4" (UID: "f0675d2f-779a-4e4d-8669-fa774841dab4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:32:31.509613 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:31.509519 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0675d2f-779a-4e4d-8669-fa774841dab4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f0675d2f-779a-4e4d-8669-fa774841dab4" (UID: "f0675d2f-779a-4e4d-8669-fa774841dab4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:32:31.509794 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:31.509746 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0675d2f-779a-4e4d-8669-fa774841dab4-istio-data" (OuterVolumeSpecName: "istio-data") pod "f0675d2f-779a-4e4d-8669-fa774841dab4" (UID: "f0675d2f-779a-4e4d-8669-fa774841dab4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:32:31.509886 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:31.509859 2546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0675d2f-779a-4e4d-8669-fa774841dab4-workload-socket\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:32:31.509886 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:31.509886 2546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0675d2f-779a-4e4d-8669-fa774841dab4-istiod-ca-cert\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:32:31.510082 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:31.509900 2546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0675d2f-779a-4e4d-8669-fa774841dab4-workload-certs\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:32:31.510082 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:31.509909 2546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0675d2f-779a-4e4d-8669-fa774841dab4-credential-socket\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:32:31.510082 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:31.509917 2546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0675d2f-779a-4e4d-8669-fa774841dab4-istio-data\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:32:31.511267 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:31.511236 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0675d2f-779a-4e4d-8669-fa774841dab4-istio-token" (OuterVolumeSpecName: "istio-token") pod "f0675d2f-779a-4e4d-8669-fa774841dab4" (UID: "f0675d2f-779a-4e4d-8669-fa774841dab4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:32:31.511368 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:31.511305 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0675d2f-779a-4e4d-8669-fa774841dab4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f0675d2f-779a-4e4d-8669-fa774841dab4" (UID: "f0675d2f-779a-4e4d-8669-fa774841dab4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 20:32:31.511513 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:31.511491 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0675d2f-779a-4e4d-8669-fa774841dab4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f0675d2f-779a-4e4d-8669-fa774841dab4" (UID: "f0675d2f-779a-4e4d-8669-fa774841dab4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:32:31.511614 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:31.511526 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0675d2f-779a-4e4d-8669-fa774841dab4-kube-api-access-sdnpk" (OuterVolumeSpecName: "kube-api-access-sdnpk") pod "f0675d2f-779a-4e4d-8669-fa774841dab4" (UID: "f0675d2f-779a-4e4d-8669-fa774841dab4"). InnerVolumeSpecName "kube-api-access-sdnpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:32:31.612262 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:31.612226 2546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0675d2f-779a-4e4d-8669-fa774841dab4-istio-envoy\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:32:31.612262 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:31.612262 2546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0675d2f-779a-4e4d-8669-fa774841dab4-istio-token\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:32:31.612469 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:31.612276 2546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0675d2f-779a-4e4d-8669-fa774841dab4-istio-podinfo\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:32:31.612469 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:31.612289 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdnpk\" (UniqueName: \"kubernetes.io/projected/f0675d2f-779a-4e4d-8669-fa774841dab4-kube-api-access-sdnpk\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:32:31.887900 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:31.885423 2546 generic.go:356] "Generic (PLEG): container finished" podID="f0675d2f-779a-4e4d-8669-fa774841dab4" containerID="49a6942e3a273eae6984711f7a933e7c1caef6df59afc8636cc1997a678bbe8d" exitCode=0 Sep 11 20:32:31.887900 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:31.885469 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x" event={"ID":"f0675d2f-779a-4e4d-8669-fa774841dab4","Type":"ContainerDied","Data":"49a6942e3a273eae6984711f7a933e7c1caef6df59afc8636cc1997a678bbe8d"} Sep 11 20:32:31.887900 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:31.885515 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x" event={"ID":"f0675d2f-779a-4e4d-8669-fa774841dab4","Type":"ContainerDied","Data":"4d14d89514b34bfee6033c35589e605b439909c359a21e32509945c1dc8654a1"} Sep 11 20:32:31.888200 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:31.887944 2546 scope.go:117] "RemoveContainer" containerID="49a6942e3a273eae6984711f7a933e7c1caef6df59afc8636cc1997a678bbe8d" Sep 11 20:32:31.888200 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:31.887983 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x" Sep 11 20:32:31.898638 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:31.898621 2546 scope.go:117] "RemoveContainer" containerID="eea6ff6be95f0d3b57a81db6cb97d410bf70cd2108448e8764bcac1c927c36cf" Sep 11 20:32:31.906536 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:31.906477 2546 scope.go:117] "RemoveContainer" containerID="49a6942e3a273eae6984711f7a933e7c1caef6df59afc8636cc1997a678bbe8d" Sep 11 20:32:31.907322 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:32:31.907296 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49a6942e3a273eae6984711f7a933e7c1caef6df59afc8636cc1997a678bbe8d\": container with ID starting with 49a6942e3a273eae6984711f7a933e7c1caef6df59afc8636cc1997a678bbe8d not found: ID does not exist" containerID="49a6942e3a273eae6984711f7a933e7c1caef6df59afc8636cc1997a678bbe8d" Sep 11 20:32:31.907322 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:31.907326 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49a6942e3a273eae6984711f7a933e7c1caef6df59afc8636cc1997a678bbe8d"} err="failed to get container status \"49a6942e3a273eae6984711f7a933e7c1caef6df59afc8636cc1997a678bbe8d\": rpc error: code = NotFound desc = could not find container \"49a6942e3a273eae6984711f7a933e7c1caef6df59afc8636cc1997a678bbe8d\": container with ID starting with 49a6942e3a273eae6984711f7a933e7c1caef6df59afc8636cc1997a678bbe8d not found: ID does not exist" Sep 11 20:32:31.907322 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:31.907342 2546 scope.go:117] "RemoveContainer" containerID="eea6ff6be95f0d3b57a81db6cb97d410bf70cd2108448e8764bcac1c927c36cf" Sep 11 20:32:31.907827 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:32:31.907787 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eea6ff6be95f0d3b57a81db6cb97d410bf70cd2108448e8764bcac1c927c36cf\": container with ID starting with eea6ff6be95f0d3b57a81db6cb97d410bf70cd2108448e8764bcac1c927c36cf not found: ID does not exist" containerID="eea6ff6be95f0d3b57a81db6cb97d410bf70cd2108448e8764bcac1c927c36cf" Sep 11 20:32:31.907939 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:31.907824 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eea6ff6be95f0d3b57a81db6cb97d410bf70cd2108448e8764bcac1c927c36cf"} err="failed to get container status \"eea6ff6be95f0d3b57a81db6cb97d410bf70cd2108448e8764bcac1c927c36cf\": rpc error: code = NotFound desc = could not find container \"eea6ff6be95f0d3b57a81db6cb97d410bf70cd2108448e8764bcac1c927c36cf\": container with ID starting with eea6ff6be95f0d3b57a81db6cb97d410bf70cd2108448e8764bcac1c927c36cf not found: ID does not exist" Sep 11 20:32:31.910409 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:31.910388 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x"] Sep 11 20:32:31.913483 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:31.913460 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-5de5ea-openshift-default-656cf5cdc4-jfc4x"] Sep 11 20:32:32.105398 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:32.105371 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:32:32.451298 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:32.451262 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0675d2f-779a-4e4d-8669-fa774841dab4" path="/var/lib/kubelet/pods/f0675d2f-779a-4e4d-8669-fa774841dab4/volumes" Sep 11 20:32:32.904037 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:32.904005 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:32:33.552706 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:33.552673 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:32:45.281597 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:45.280455 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-248ab6-8695cbc44c-qgrcz"] Sep 11 20:32:45.281597 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:45.280767 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-248ab6-8695cbc44c-qgrcz" podUID="d6477778-0da8-45d2-bc2f-222e7f7a9d2f" containerName="maas-api" containerID="cri-o://dc95c584fc3ba3bd97ce60881c00fc7b7fd30021b0ed08e0db446ec6fdcef23a" gracePeriod=30 Sep 11 20:32:45.310598 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:45.310540 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:32:45.341649 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:45.341619 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-248ab6-6dcbcf4bc8-qsqxb"] Sep 11 20:32:45.341883 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:45.341860 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-248ab6-6dcbcf4bc8-qsqxb" podUID="3794824d-d5d9-4f3a-bcda-75e80942a10a" containerName="payload-pre-processing" containerID="cri-o://5b26e711eba3d21b94bcbeb4f34f7e61f7dc4172e3e2a553800ee7374b1dc866" gracePeriod=30 Sep 11 20:32:45.376559 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:45.376525 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:32:45.476951 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:45.476918 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:32:45.533469 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:45.533406 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:32:45.612907 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:45.612875 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-248ab6-8695cbc44c-qgrcz" Sep 11 20:32:45.641863 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:45.641841 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-248ab6-6dcbcf4bc8-qsqxb" Sep 11 20:32:45.643657 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:45.643637 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6477778-0da8-45d2-bc2f-222e7f7a9d2f-metrics-certs\") pod \"d6477778-0da8-45d2-bc2f-222e7f7a9d2f\" (UID: \"d6477778-0da8-45d2-bc2f-222e7f7a9d2f\") " Sep 11 20:32:45.643771 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:45.643681 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nwgw6\" (UniqueName: \"kubernetes.io/projected/d6477778-0da8-45d2-bc2f-222e7f7a9d2f-kube-api-access-nwgw6\") pod \"d6477778-0da8-45d2-bc2f-222e7f7a9d2f\" (UID: \"d6477778-0da8-45d2-bc2f-222e7f7a9d2f\") " Sep 11 20:32:45.643837 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:45.643823 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6477778-0da8-45d2-bc2f-222e7f7a9d2f-openshift-service-ca\") pod \"d6477778-0da8-45d2-bc2f-222e7f7a9d2f\" (UID: \"d6477778-0da8-45d2-bc2f-222e7f7a9d2f\") " Sep 11 20:32:45.643894 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:45.643861 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6477778-0da8-45d2-bc2f-222e7f7a9d2f-maas-api-tls\") pod \"d6477778-0da8-45d2-bc2f-222e7f7a9d2f\" (UID: \"d6477778-0da8-45d2-bc2f-222e7f7a9d2f\") " Sep 11 20:32:45.644215 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:45.644156 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6477778-0da8-45d2-bc2f-222e7f7a9d2f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d6477778-0da8-45d2-bc2f-222e7f7a9d2f" (UID: "d6477778-0da8-45d2-bc2f-222e7f7a9d2f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:32:45.644355 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:45.644335 2546 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6477778-0da8-45d2-bc2f-222e7f7a9d2f-openshift-service-ca\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:32:45.645990 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:45.645952 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6477778-0da8-45d2-bc2f-222e7f7a9d2f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d6477778-0da8-45d2-bc2f-222e7f7a9d2f" (UID: "d6477778-0da8-45d2-bc2f-222e7f7a9d2f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 20:32:45.646108 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:45.646048 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6477778-0da8-45d2-bc2f-222e7f7a9d2f-kube-api-access-nwgw6" (OuterVolumeSpecName: "kube-api-access-nwgw6") pod "d6477778-0da8-45d2-bc2f-222e7f7a9d2f" (UID: "d6477778-0da8-45d2-bc2f-222e7f7a9d2f"). InnerVolumeSpecName "kube-api-access-nwgw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:32:45.646283 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:45.646265 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6477778-0da8-45d2-bc2f-222e7f7a9d2f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d6477778-0da8-45d2-bc2f-222e7f7a9d2f" (UID: "d6477778-0da8-45d2-bc2f-222e7f7a9d2f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 20:32:45.745006 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:45.744967 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3794824d-d5d9-4f3a-bcda-75e80942a10a-plugins-config-volume\") pod \"3794824d-d5d9-4f3a-bcda-75e80942a10a\" (UID: \"3794824d-d5d9-4f3a-bcda-75e80942a10a\") " Sep 11 20:32:45.745173 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:45.745029 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vmpl2\" (UniqueName: \"kubernetes.io/projected/3794824d-d5d9-4f3a-bcda-75e80942a10a-kube-api-access-vmpl2\") pod \"3794824d-d5d9-4f3a-bcda-75e80942a10a\" (UID: \"3794824d-d5d9-4f3a-bcda-75e80942a10a\") " Sep 11 20:32:45.745328 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:45.745303 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3794824d-d5d9-4f3a-bcda-75e80942a10a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3794824d-d5d9-4f3a-bcda-75e80942a10a" (UID: "3794824d-d5d9-4f3a-bcda-75e80942a10a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:32:45.745408 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:45.745395 2546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3794824d-d5d9-4f3a-bcda-75e80942a10a-plugins-config-volume\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:32:45.745466 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:45.745410 2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6477778-0da8-45d2-bc2f-222e7f7a9d2f-maas-api-tls\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:32:45.745466 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:45.745420 2546 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6477778-0da8-45d2-bc2f-222e7f7a9d2f-metrics-certs\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:32:45.745466 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:45.745428 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwgw6\" (UniqueName: \"kubernetes.io/projected/d6477778-0da8-45d2-bc2f-222e7f7a9d2f-kube-api-access-nwgw6\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:32:45.747095 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:45.747061 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3794824d-d5d9-4f3a-bcda-75e80942a10a-kube-api-access-vmpl2" (OuterVolumeSpecName: "kube-api-access-vmpl2") pod "3794824d-d5d9-4f3a-bcda-75e80942a10a" (UID: "3794824d-d5d9-4f3a-bcda-75e80942a10a"). InnerVolumeSpecName "kube-api-access-vmpl2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:32:45.846070 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:45.846041 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmpl2\" (UniqueName: \"kubernetes.io/projected/3794824d-d5d9-4f3a-bcda-75e80942a10a-kube-api-access-vmpl2\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:32:45.859120 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:45.859090 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:32:45.959489 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:45.959455 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:32:45.962534 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:45.962505 2546 generic.go:356] "Generic (PLEG): container finished" podID="d6477778-0da8-45d2-bc2f-222e7f7a9d2f" containerID="dc95c584fc3ba3bd97ce60881c00fc7b7fd30021b0ed08e0db446ec6fdcef23a" exitCode=0 Sep 11 20:32:45.962684 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:45.962553 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-248ab6-8695cbc44c-qgrcz" Sep 11 20:32:45.962684 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:45.962614 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-248ab6-8695cbc44c-qgrcz" event={"ID":"d6477778-0da8-45d2-bc2f-222e7f7a9d2f","Type":"ContainerDied","Data":"dc95c584fc3ba3bd97ce60881c00fc7b7fd30021b0ed08e0db446ec6fdcef23a"} Sep 11 20:32:45.962684 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:45.962668 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-248ab6-8695cbc44c-qgrcz" event={"ID":"d6477778-0da8-45d2-bc2f-222e7f7a9d2f","Type":"ContainerDied","Data":"499858ee3ff2c27cdf303f36aa048bcb8e1fd91a757aedc65b02f6e6a1db7913"} Sep 11 20:32:45.962814 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:45.962701 2546 scope.go:117] "RemoveContainer" containerID="dc95c584fc3ba3bd97ce60881c00fc7b7fd30021b0ed08e0db446ec6fdcef23a" Sep 11 20:32:45.964095 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:45.964057 2546 generic.go:356] "Generic (PLEG): container finished" podID="3794824d-d5d9-4f3a-bcda-75e80942a10a" containerID="5b26e711eba3d21b94bcbeb4f34f7e61f7dc4172e3e2a553800ee7374b1dc866" exitCode=0 Sep 11 20:32:45.964196 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:45.964106 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-248ab6-6dcbcf4bc8-qsqxb" Sep 11 20:32:45.964196 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:45.964110 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-248ab6-6dcbcf4bc8-qsqxb" event={"ID":"3794824d-d5d9-4f3a-bcda-75e80942a10a","Type":"ContainerDied","Data":"5b26e711eba3d21b94bcbeb4f34f7e61f7dc4172e3e2a553800ee7374b1dc866"} Sep 11 20:32:45.964196 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:45.964162 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-248ab6-6dcbcf4bc8-qsqxb" event={"ID":"3794824d-d5d9-4f3a-bcda-75e80942a10a","Type":"ContainerDied","Data":"33c81ea93c5ade78463abca6e7d0e88f288862682150d8e3f26e3abe0b97a263"} Sep 11 20:32:45.973109 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:45.973092 2546 scope.go:117] "RemoveContainer" containerID="dc95c584fc3ba3bd97ce60881c00fc7b7fd30021b0ed08e0db446ec6fdcef23a" Sep 11 20:32:45.973363 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:32:45.973340 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc95c584fc3ba3bd97ce60881c00fc7b7fd30021b0ed08e0db446ec6fdcef23a\": container with ID starting with dc95c584fc3ba3bd97ce60881c00fc7b7fd30021b0ed08e0db446ec6fdcef23a not found: ID does not exist" containerID="dc95c584fc3ba3bd97ce60881c00fc7b7fd30021b0ed08e0db446ec6fdcef23a" Sep 11 20:32:45.973428 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:45.973379 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc95c584fc3ba3bd97ce60881c00fc7b7fd30021b0ed08e0db446ec6fdcef23a"} err="failed to get container status \"dc95c584fc3ba3bd97ce60881c00fc7b7fd30021b0ed08e0db446ec6fdcef23a\": rpc error: code = NotFound desc = could not find container \"dc95c584fc3ba3bd97ce60881c00fc7b7fd30021b0ed08e0db446ec6fdcef23a\": container with ID starting with dc95c584fc3ba3bd97ce60881c00fc7b7fd30021b0ed08e0db446ec6fdcef23a not found: ID does not exist" Sep 11 20:32:45.973428 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:45.973404 2546 scope.go:117] "RemoveContainer" containerID="5b26e711eba3d21b94bcbeb4f34f7e61f7dc4172e3e2a553800ee7374b1dc866" Sep 11 20:32:45.981606 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:45.981588 2546 scope.go:117] "RemoveContainer" containerID="5b26e711eba3d21b94bcbeb4f34f7e61f7dc4172e3e2a553800ee7374b1dc866" Sep 11 20:32:45.981855 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:32:45.981836 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b26e711eba3d21b94bcbeb4f34f7e61f7dc4172e3e2a553800ee7374b1dc866\": container with ID starting with 5b26e711eba3d21b94bcbeb4f34f7e61f7dc4172e3e2a553800ee7374b1dc866 not found: ID does not exist" containerID="5b26e711eba3d21b94bcbeb4f34f7e61f7dc4172e3e2a553800ee7374b1dc866" Sep 11 20:32:45.981905 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:45.981862 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b26e711eba3d21b94bcbeb4f34f7e61f7dc4172e3e2a553800ee7374b1dc866"} err="failed to get container status \"5b26e711eba3d21b94bcbeb4f34f7e61f7dc4172e3e2a553800ee7374b1dc866\": rpc error: code = NotFound desc = could not find container \"5b26e711eba3d21b94bcbeb4f34f7e61f7dc4172e3e2a553800ee7374b1dc866\": container with ID starting with 5b26e711eba3d21b94bcbeb4f34f7e61f7dc4172e3e2a553800ee7374b1dc866 not found: ID does not exist" Sep 11 20:32:45.991232 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:45.991204 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-248ab6-8695cbc44c-qgrcz"] Sep 11 20:32:45.994046 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:45.994015 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-248ab6-8695cbc44c-qgrcz"] Sep 11 20:32:46.003157 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:46.003137 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-248ab6-6dcbcf4bc8-qsqxb"] Sep 11 20:32:46.008439 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:46.008419 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-248ab6-6dcbcf4bc8-qsqxb"] Sep 11 20:32:46.451566 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:46.451531 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3794824d-d5d9-4f3a-bcda-75e80942a10a" path="/var/lib/kubelet/pods/3794824d-d5d9-4f3a-bcda-75e80942a10a/volumes" Sep 11 20:32:46.452062 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:46.452042 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6477778-0da8-45d2-bc2f-222e7f7a9d2f" path="/var/lib/kubelet/pods/d6477778-0da8-45d2-bc2f-222e7f7a9d2f/volumes" Sep 11 20:32:46.598889 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:46.598846 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:32:47.394030 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:47.393993 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:32:48.107279 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:48.107238 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:32:50.968799 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:32:50.968764 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:33:13.652420 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:33:13.652387 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:33:13.706378 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:33:13.706354 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:33:27.022670 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:33:27.022566 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:33:27.899239 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:33:27.899208 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:33:32.211411 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:33:32.211382 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:33:33.335397 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:33:33.335364 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:33:33.435858 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:33:33.435818 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:33:33.605675 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:33:33.605645 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:33:34.326635 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:33:34.326599 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:33:35.098512 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:33:35.098479 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:33:35.829950 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:33:35.829920 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:33:36.501775 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:33:36.501741 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:33:36.840293 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:33:36.840267 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:33:36.942129 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:33:36.942100 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:33:37.222102 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:33:37.222029 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:33:44.420169 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:33:44.420131 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:33:44.454239 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:33:44.454211 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b6453d-66bf6d488b-cvxbp"] Sep 11 20:33:44.454545 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:33:44.454502 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6453d-66bf6d488b-cvxbp" podUID="6cc670c4-aa51-441e-9027-68c0886aab39" containerName="payload-pre-processing" containerID="cri-o://19dd28fe4ed5ed359be2054337793baebf412af8d336bb590a272317db95858b" gracePeriod=30 Sep 11 20:33:44.520551 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:33:44.520513 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:33:44.609870 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:33:44.609837 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:33:44.672224 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:33:44.672120 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:33:44.728929 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:33:44.728893 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6453d-66bf6d488b-cvxbp" Sep 11 20:33:44.793752 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:33:44.793716 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cc670c4-aa51-441e-9027-68c0886aab39-plugins-config-volume\") pod \"6cc670c4-aa51-441e-9027-68c0886aab39\" (UID: \"6cc670c4-aa51-441e-9027-68c0886aab39\") " Sep 11 20:33:44.793954 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:33:44.793766 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65l5n\" (UniqueName: \"kubernetes.io/projected/6cc670c4-aa51-441e-9027-68c0886aab39-kube-api-access-65l5n\") pod \"6cc670c4-aa51-441e-9027-68c0886aab39\" (UID: \"6cc670c4-aa51-441e-9027-68c0886aab39\") " Sep 11 20:33:44.794171 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:33:44.794146 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cc670c4-aa51-441e-9027-68c0886aab39-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6cc670c4-aa51-441e-9027-68c0886aab39" (UID: "6cc670c4-aa51-441e-9027-68c0886aab39"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:33:44.796067 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:33:44.796041 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cc670c4-aa51-441e-9027-68c0886aab39-kube-api-access-65l5n" (OuterVolumeSpecName: "kube-api-access-65l5n") pod "6cc670c4-aa51-441e-9027-68c0886aab39" (UID: "6cc670c4-aa51-441e-9027-68c0886aab39"). InnerVolumeSpecName "kube-api-access-65l5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:33:44.895294 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:33:44.895260 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65l5n\" (UniqueName: \"kubernetes.io/projected/6cc670c4-aa51-441e-9027-68c0886aab39-kube-api-access-65l5n\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:33:44.895294 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:33:44.895290 2546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cc670c4-aa51-441e-9027-68c0886aab39-plugins-config-volume\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:33:45.045722 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:33:45.045640 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:33:45.140194 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:33:45.140153 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:33:45.283350 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:33:45.283317 2546 generic.go:356] "Generic (PLEG): container finished" podID="6cc670c4-aa51-441e-9027-68c0886aab39" containerID="19dd28fe4ed5ed359be2054337793baebf412af8d336bb590a272317db95858b" exitCode=0 Sep 11 20:33:45.283522 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:33:45.283355 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6453d-66bf6d488b-cvxbp" event={"ID":"6cc670c4-aa51-441e-9027-68c0886aab39","Type":"ContainerDied","Data":"19dd28fe4ed5ed359be2054337793baebf412af8d336bb590a272317db95858b"} Sep 11 20:33:45.283522 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:33:45.283366 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6453d-66bf6d488b-cvxbp" Sep 11 20:33:45.283522 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:33:45.283433 2546 scope.go:117] "RemoveContainer" containerID="19dd28fe4ed5ed359be2054337793baebf412af8d336bb590a272317db95858b" Sep 11 20:33:45.283522 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:33:45.283422 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6453d-66bf6d488b-cvxbp" event={"ID":"6cc670c4-aa51-441e-9027-68c0886aab39","Type":"ContainerDied","Data":"ee80475fa8869b0173fb5700fb4c4578ef8311f382e2137c0065bef938d2b91f"} Sep 11 20:33:45.292387 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:33:45.292370 2546 scope.go:117] "RemoveContainer" containerID="19dd28fe4ed5ed359be2054337793baebf412af8d336bb590a272317db95858b" Sep 11 20:33:45.292695 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:33:45.292672 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19dd28fe4ed5ed359be2054337793baebf412af8d336bb590a272317db95858b\": container with ID starting with 19dd28fe4ed5ed359be2054337793baebf412af8d336bb590a272317db95858b not found: ID does not exist" containerID="19dd28fe4ed5ed359be2054337793baebf412af8d336bb590a272317db95858b" Sep 11 20:33:45.292779 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:33:45.292706 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19dd28fe4ed5ed359be2054337793baebf412af8d336bb590a272317db95858b"} err="failed to get container status \"19dd28fe4ed5ed359be2054337793baebf412af8d336bb590a272317db95858b\": rpc error: code = NotFound desc = could not find container \"19dd28fe4ed5ed359be2054337793baebf412af8d336bb590a272317db95858b\": container with ID starting with 19dd28fe4ed5ed359be2054337793baebf412af8d336bb590a272317db95858b not found: ID does not exist" Sep 11 20:33:45.318407 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:33:45.316342 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b6453d-66bf6d488b-cvxbp"] Sep 11 20:33:45.318407 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:33:45.316395 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b6453d-66bf6d488b-cvxbp"] Sep 11 20:33:45.913280 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:33:45.913249 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:33:46.452216 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:33:46.452180 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6cc670c4-aa51-441e-9027-68c0886aab39" path="/var/lib/kubelet/pods/6cc670c4-aa51-441e-9027-68c0886aab39/volumes" Sep 11 20:33:46.560270 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:33:46.560239 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:33:50.675546 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:33:50.675512 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:33:50.776656 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:33:50.776625 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:33:50.983724 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:33:50.983650 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:33:51.527382 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:33:51.527347 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:33:52.314315 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:33:52.314274 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:33:52.947538 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:33:52.947502 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:33:53.605511 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:33:53.605481 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:34:01.708857 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:01.708821 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:34:01.734675 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:01.734638 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-fcb0e4-5497ffdf9-9zjdw"] Sep 11 20:34:01.734951 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:01.734897 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fcb0e4-5497ffdf9-9zjdw" podUID="461ac6f0-a3e8-4526-849a-290a9872e785" containerName="payload-processing" containerID="cri-o://37bb93cb777e95a16f4d6cbabf08a048099f5df5ce2f7e391525bb25193974e7" gracePeriod=30 Sep 11 20:34:01.741276 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:01.741119 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fcb0e4-755659c658-t9zvt"] Sep 11 20:34:01.741828 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:01.741331 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fcb0e4-755659c658-t9zvt" podUID="e34b67c7-d6e4-4819-a353-76004387dc4b" containerName="payload-pre-processing" containerID="cri-o://8c23f22f991afcd2af236a4cef804f0cf6f16501e4d6df2e678ad51f41ec95ab" gracePeriod=30 Sep 11 20:34:01.780322 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:01.780285 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:34:01.837403 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:01.837350 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:34:01.897815 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:01.897785 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:34:01.956091 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:01.956063 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:34:02.020660 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:02.020636 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fcb0e4-5497ffdf9-9zjdw" Sep 11 20:34:02.023628 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:02.023610 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fcb0e4-755659c658-t9zvt" Sep 11 20:34:02.164862 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:02.164828 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fm9bt\" (UniqueName: \"kubernetes.io/projected/e34b67c7-d6e4-4819-a353-76004387dc4b-kube-api-access-fm9bt\") pod \"e34b67c7-d6e4-4819-a353-76004387dc4b\" (UID: \"e34b67c7-d6e4-4819-a353-76004387dc4b\") " Sep 11 20:34:02.165025 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:02.164873 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/461ac6f0-a3e8-4526-849a-290a9872e785-plugins-config-volume\") pod \"461ac6f0-a3e8-4526-849a-290a9872e785\" (UID: \"461ac6f0-a3e8-4526-849a-290a9872e785\") " Sep 11 20:34:02.165025 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:02.164925 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lvk9b\" (UniqueName: \"kubernetes.io/projected/461ac6f0-a3e8-4526-849a-290a9872e785-kube-api-access-lvk9b\") pod \"461ac6f0-a3e8-4526-849a-290a9872e785\" (UID: \"461ac6f0-a3e8-4526-849a-290a9872e785\") " Sep 11 20:34:02.165025 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:02.165005 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e34b67c7-d6e4-4819-a353-76004387dc4b-plugins-config-volume\") pod \"e34b67c7-d6e4-4819-a353-76004387dc4b\" (UID: \"e34b67c7-d6e4-4819-a353-76004387dc4b\") " Sep 11 20:34:02.165385 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:02.165353 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/461ac6f0-a3e8-4526-849a-290a9872e785-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "461ac6f0-a3e8-4526-849a-290a9872e785" (UID: "461ac6f0-a3e8-4526-849a-290a9872e785"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:34:02.165505 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:02.165391 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e34b67c7-d6e4-4819-a353-76004387dc4b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e34b67c7-d6e4-4819-a353-76004387dc4b" (UID: "e34b67c7-d6e4-4819-a353-76004387dc4b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:34:02.166960 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:02.166939 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e34b67c7-d6e4-4819-a353-76004387dc4b-kube-api-access-fm9bt" (OuterVolumeSpecName: "kube-api-access-fm9bt") pod "e34b67c7-d6e4-4819-a353-76004387dc4b" (UID: "e34b67c7-d6e4-4819-a353-76004387dc4b"). InnerVolumeSpecName "kube-api-access-fm9bt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:34:02.167092 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:02.167072 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/461ac6f0-a3e8-4526-849a-290a9872e785-kube-api-access-lvk9b" (OuterVolumeSpecName: "kube-api-access-lvk9b") pod "461ac6f0-a3e8-4526-849a-290a9872e785" (UID: "461ac6f0-a3e8-4526-849a-290a9872e785"). InnerVolumeSpecName "kube-api-access-lvk9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:34:02.246412 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:02.246334 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:34:02.266319 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:02.266295 2546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e34b67c7-d6e4-4819-a353-76004387dc4b-plugins-config-volume\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:34:02.266319 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:02.266316 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm9bt\" (UniqueName: \"kubernetes.io/projected/e34b67c7-d6e4-4819-a353-76004387dc4b-kube-api-access-fm9bt\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:34:02.266467 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:02.266326 2546 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/461ac6f0-a3e8-4526-849a-290a9872e785-plugins-config-volume\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:34:02.266467 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:02.266336 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvk9b\" (UniqueName: \"kubernetes.io/projected/461ac6f0-a3e8-4526-849a-290a9872e785-kube-api-access-lvk9b\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:34:02.346849 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:02.346817 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:34:02.383225 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:02.383197 2546 generic.go:356] "Generic (PLEG): container finished" podID="461ac6f0-a3e8-4526-849a-290a9872e785" containerID="37bb93cb777e95a16f4d6cbabf08a048099f5df5ce2f7e391525bb25193974e7" exitCode=0 Sep 11 20:34:02.383366 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:02.383230 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fcb0e4-5497ffdf9-9zjdw" event={"ID":"461ac6f0-a3e8-4526-849a-290a9872e785","Type":"ContainerDied","Data":"37bb93cb777e95a16f4d6cbabf08a048099f5df5ce2f7e391525bb25193974e7"} Sep 11 20:34:02.383366 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:02.383260 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fcb0e4-5497ffdf9-9zjdw" Sep 11 20:34:02.383366 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:02.383306 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fcb0e4-5497ffdf9-9zjdw" event={"ID":"461ac6f0-a3e8-4526-849a-290a9872e785","Type":"ContainerDied","Data":"b95f391f0b9d67f9136fe8acf7be5fdca2f02fd9628a0d930cc2cc5856e349f3"} Sep 11 20:34:02.383366 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:02.383340 2546 scope.go:117] "RemoveContainer" containerID="37bb93cb777e95a16f4d6cbabf08a048099f5df5ce2f7e391525bb25193974e7" Sep 11 20:34:02.386244 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:02.386204 2546 generic.go:356] "Generic (PLEG): container finished" podID="e34b67c7-d6e4-4819-a353-76004387dc4b" containerID="8c23f22f991afcd2af236a4cef804f0cf6f16501e4d6df2e678ad51f41ec95ab" exitCode=0 Sep 11 20:34:02.386244 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:02.386238 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fcb0e4-755659c658-t9zvt" Sep 11 20:34:02.386422 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:02.386258 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fcb0e4-755659c658-t9zvt" event={"ID":"e34b67c7-d6e4-4819-a353-76004387dc4b","Type":"ContainerDied","Data":"8c23f22f991afcd2af236a4cef804f0cf6f16501e4d6df2e678ad51f41ec95ab"} Sep 11 20:34:02.386422 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:02.386301 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fcb0e4-755659c658-t9zvt" event={"ID":"e34b67c7-d6e4-4819-a353-76004387dc4b","Type":"ContainerDied","Data":"7900c6390960ee14bc20c1a8bdb8a4d02cb0aca474a410408062f44d0f7f8523"} Sep 11 20:34:02.397456 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:02.397433 2546 scope.go:117] "RemoveContainer" containerID="37bb93cb777e95a16f4d6cbabf08a048099f5df5ce2f7e391525bb25193974e7" Sep 11 20:34:02.397842 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:34:02.397817 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37bb93cb777e95a16f4d6cbabf08a048099f5df5ce2f7e391525bb25193974e7\": container with ID starting with 37bb93cb777e95a16f4d6cbabf08a048099f5df5ce2f7e391525bb25193974e7 not found: ID does not exist" containerID="37bb93cb777e95a16f4d6cbabf08a048099f5df5ce2f7e391525bb25193974e7" Sep 11 20:34:02.397949 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:02.397849 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37bb93cb777e95a16f4d6cbabf08a048099f5df5ce2f7e391525bb25193974e7"} err="failed to get container status \"37bb93cb777e95a16f4d6cbabf08a048099f5df5ce2f7e391525bb25193974e7\": rpc error: code = NotFound desc = could not find container \"37bb93cb777e95a16f4d6cbabf08a048099f5df5ce2f7e391525bb25193974e7\": container with ID starting with 37bb93cb777e95a16f4d6cbabf08a048099f5df5ce2f7e391525bb25193974e7 not found: ID does not exist" Sep 11 20:34:02.397949 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:02.397871 2546 scope.go:117] "RemoveContainer" containerID="8c23f22f991afcd2af236a4cef804f0cf6f16501e4d6df2e678ad51f41ec95ab" Sep 11 20:34:02.405846 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:02.405829 2546 scope.go:117] "RemoveContainer" containerID="8c23f22f991afcd2af236a4cef804f0cf6f16501e4d6df2e678ad51f41ec95ab" Sep 11 20:34:02.406123 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:34:02.406101 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c23f22f991afcd2af236a4cef804f0cf6f16501e4d6df2e678ad51f41ec95ab\": container with ID starting with 8c23f22f991afcd2af236a4cef804f0cf6f16501e4d6df2e678ad51f41ec95ab not found: ID does not exist" containerID="8c23f22f991afcd2af236a4cef804f0cf6f16501e4d6df2e678ad51f41ec95ab" Sep 11 20:34:02.406204 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:02.406132 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c23f22f991afcd2af236a4cef804f0cf6f16501e4d6df2e678ad51f41ec95ab"} err="failed to get container status \"8c23f22f991afcd2af236a4cef804f0cf6f16501e4d6df2e678ad51f41ec95ab\": rpc error: code = NotFound desc = could not find container \"8c23f22f991afcd2af236a4cef804f0cf6f16501e4d6df2e678ad51f41ec95ab\": container with ID starting with 8c23f22f991afcd2af236a4cef804f0cf6f16501e4d6df2e678ad51f41ec95ab not found: ID does not exist" Sep 11 20:34:02.408661 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:02.408639 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-fcb0e4-5497ffdf9-9zjdw"] Sep 11 20:34:02.412798 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:02.412779 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-fcb0e4-5497ffdf9-9zjdw"] Sep 11 20:34:02.421199 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:02.421180 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fcb0e4-755659c658-t9zvt"] Sep 11 20:34:02.422715 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:02.422697 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fcb0e4-755659c658-t9zvt"] Sep 11 20:34:02.449610 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:02.449567 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="461ac6f0-a3e8-4526-849a-290a9872e785" path="/var/lib/kubelet/pods/461ac6f0-a3e8-4526-849a-290a9872e785/volumes" Sep 11 20:34:02.450031 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:02.450013 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e34b67c7-d6e4-4819-a353-76004387dc4b" path="/var/lib/kubelet/pods/e34b67c7-d6e4-4819-a353-76004387dc4b/volumes" Sep 11 20:34:03.070137 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:03.070101 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:34:07.914554 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:07.914518 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:34:07.971034 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:07.971000 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:34:08.138766 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:08.138733 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s"] Sep 11 20:34:08.139002 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:08.138974 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s" podUID="b559f06b-d917-4fb4-a548-31e824e58070" containerName="istio-proxy" containerID="cri-o://beb00cb9814cda29b18dd566fd38f5b148027cf79b3b6c1e64c1f2fecad5f977" gracePeriod=30 Sep 11 20:34:08.159817 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:08.159788 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:34:09.321939 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:09.321899 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:34:10.049403 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:10.049371 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:34:13.398188 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:13.398162 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s" Sep 11 20:34:13.446551 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:13.446475 2546 generic.go:356] "Generic (PLEG): container finished" podID="b559f06b-d917-4fb4-a548-31e824e58070" containerID="beb00cb9814cda29b18dd566fd38f5b148027cf79b3b6c1e64c1f2fecad5f977" exitCode=0 Sep 11 20:34:13.446551 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:13.446529 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s" Sep 11 20:34:13.446773 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:13.446551 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s" event={"ID":"b559f06b-d917-4fb4-a548-31e824e58070","Type":"ContainerDied","Data":"beb00cb9814cda29b18dd566fd38f5b148027cf79b3b6c1e64c1f2fecad5f977"} Sep 11 20:34:13.446773 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:13.446620 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s" event={"ID":"b559f06b-d917-4fb4-a548-31e824e58070","Type":"ContainerDied","Data":"21cf8f2cc94d2840fe88a689cd01f16c49fcc03893b1a063dd89f06ceb49ccb2"} Sep 11 20:34:13.446773 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:13.446646 2546 scope.go:117] "RemoveContainer" containerID="beb00cb9814cda29b18dd566fd38f5b148027cf79b3b6c1e64c1f2fecad5f977" Sep 11 20:34:13.454806 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:13.454791 2546 scope.go:117] "RemoveContainer" containerID="48b40b50bb4ec3d4dd84e5f88be94d0f05deb69df33942df7c6b8cbd95b7de93" Sep 11 20:34:13.462226 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:13.462207 2546 scope.go:117] "RemoveContainer" containerID="beb00cb9814cda29b18dd566fd38f5b148027cf79b3b6c1e64c1f2fecad5f977" Sep 11 20:34:13.462515 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:34:13.462497 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"beb00cb9814cda29b18dd566fd38f5b148027cf79b3b6c1e64c1f2fecad5f977\": container with ID starting with beb00cb9814cda29b18dd566fd38f5b148027cf79b3b6c1e64c1f2fecad5f977 not found: ID does not exist" containerID="beb00cb9814cda29b18dd566fd38f5b148027cf79b3b6c1e64c1f2fecad5f977" Sep 11 20:34:13.462561 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:13.462524 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"beb00cb9814cda29b18dd566fd38f5b148027cf79b3b6c1e64c1f2fecad5f977"} err="failed to get container status \"beb00cb9814cda29b18dd566fd38f5b148027cf79b3b6c1e64c1f2fecad5f977\": rpc error: code = NotFound desc = could not find container \"beb00cb9814cda29b18dd566fd38f5b148027cf79b3b6c1e64c1f2fecad5f977\": container with ID starting with beb00cb9814cda29b18dd566fd38f5b148027cf79b3b6c1e64c1f2fecad5f977 not found: ID does not exist" Sep 11 20:34:13.462561 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:13.462540 2546 scope.go:117] "RemoveContainer" containerID="48b40b50bb4ec3d4dd84e5f88be94d0f05deb69df33942df7c6b8cbd95b7de93" Sep 11 20:34:13.462829 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:34:13.462812 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48b40b50bb4ec3d4dd84e5f88be94d0f05deb69df33942df7c6b8cbd95b7de93\": container with ID starting with 48b40b50bb4ec3d4dd84e5f88be94d0f05deb69df33942df7c6b8cbd95b7de93 not found: ID does not exist" containerID="48b40b50bb4ec3d4dd84e5f88be94d0f05deb69df33942df7c6b8cbd95b7de93" Sep 11 20:34:13.462887 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:13.462838 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48b40b50bb4ec3d4dd84e5f88be94d0f05deb69df33942df7c6b8cbd95b7de93"} err="failed to get container status \"48b40b50bb4ec3d4dd84e5f88be94d0f05deb69df33942df7c6b8cbd95b7de93\": rpc error: code = NotFound desc = could not find container \"48b40b50bb4ec3d4dd84e5f88be94d0f05deb69df33942df7c6b8cbd95b7de93\": container with ID starting with 48b40b50bb4ec3d4dd84e5f88be94d0f05deb69df33942df7c6b8cbd95b7de93 not found: ID does not exist" Sep 11 20:34:13.471201 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:13.471184 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b559f06b-d917-4fb4-a548-31e824e58070-istio-envoy\") pod \"b559f06b-d917-4fb4-a548-31e824e58070\" (UID: \"b559f06b-d917-4fb4-a548-31e824e58070\") " Sep 11 20:34:13.471286 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:13.471216 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b559f06b-d917-4fb4-a548-31e824e58070-istio-data\") pod \"b559f06b-d917-4fb4-a548-31e824e58070\" (UID: \"b559f06b-d917-4fb4-a548-31e824e58070\") " Sep 11 20:34:13.471346 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:13.471290 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2nfkm\" (UniqueName: \"kubernetes.io/projected/b559f06b-d917-4fb4-a548-31e824e58070-kube-api-access-2nfkm\") pod \"b559f06b-d917-4fb4-a548-31e824e58070\" (UID: \"b559f06b-d917-4fb4-a548-31e824e58070\") " Sep 11 20:34:13.471403 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:13.471345 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b559f06b-d917-4fb4-a548-31e824e58070-istio-token\") pod \"b559f06b-d917-4fb4-a548-31e824e58070\" (UID: \"b559f06b-d917-4fb4-a548-31e824e58070\") " Sep 11 20:34:13.471403 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:13.471380 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b559f06b-d917-4fb4-a548-31e824e58070-istio-podinfo\") pod \"b559f06b-d917-4fb4-a548-31e824e58070\" (UID: \"b559f06b-d917-4fb4-a548-31e824e58070\") " Sep 11 20:34:13.471505 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:13.471486 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b559f06b-d917-4fb4-a548-31e824e58070-istio-data" (OuterVolumeSpecName: "istio-data") pod "b559f06b-d917-4fb4-a548-31e824e58070" (UID: "b559f06b-d917-4fb4-a548-31e824e58070"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:34:13.471561 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:13.471496 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b559f06b-d917-4fb4-a548-31e824e58070-istiod-ca-cert\") pod \"b559f06b-d917-4fb4-a548-31e824e58070\" (UID: \"b559f06b-d917-4fb4-a548-31e824e58070\") " Sep 11 20:34:13.471630 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:13.471570 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b559f06b-d917-4fb4-a548-31e824e58070-workload-certs\") pod \"b559f06b-d917-4fb4-a548-31e824e58070\" (UID: \"b559f06b-d917-4fb4-a548-31e824e58070\") " Sep 11 20:34:13.471630 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:13.471621 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b559f06b-d917-4fb4-a548-31e824e58070-credential-socket\") pod \"b559f06b-d917-4fb4-a548-31e824e58070\" (UID: \"b559f06b-d917-4fb4-a548-31e824e58070\") " Sep 11 20:34:13.471747 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:13.471666 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b559f06b-d917-4fb4-a548-31e824e58070-workload-socket\") pod \"b559f06b-d917-4fb4-a548-31e824e58070\" (UID: \"b559f06b-d917-4fb4-a548-31e824e58070\") " Sep 11 20:34:13.472192 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:13.471966 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b559f06b-d917-4fb4-a548-31e824e58070-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b559f06b-d917-4fb4-a548-31e824e58070" (UID: "b559f06b-d917-4fb4-a548-31e824e58070"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:34:13.472192 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:13.471892 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b559f06b-d917-4fb4-a548-31e824e58070-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b559f06b-d917-4fb4-a548-31e824e58070" (UID: "b559f06b-d917-4fb4-a548-31e824e58070"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:34:13.472192 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:13.472092 2546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b559f06b-d917-4fb4-a548-31e824e58070-credential-socket\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:34:13.472364 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:13.472208 2546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b559f06b-d917-4fb4-a548-31e824e58070-istio-data\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:34:13.472364 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:13.472157 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b559f06b-d917-4fb4-a548-31e824e58070-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b559f06b-d917-4fb4-a548-31e824e58070" (UID: "b559f06b-d917-4fb4-a548-31e824e58070"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:34:13.472364 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:13.472173 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b559f06b-d917-4fb4-a548-31e824e58070-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b559f06b-d917-4fb4-a548-31e824e58070" (UID: "b559f06b-d917-4fb4-a548-31e824e58070"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:34:13.473314 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:13.473290 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b559f06b-d917-4fb4-a548-31e824e58070-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b559f06b-d917-4fb4-a548-31e824e58070" (UID: "b559f06b-d917-4fb4-a548-31e824e58070"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:34:13.473513 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:13.473489 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b559f06b-d917-4fb4-a548-31e824e58070-istio-token" (OuterVolumeSpecName: "istio-token") pod "b559f06b-d917-4fb4-a548-31e824e58070" (UID: "b559f06b-d917-4fb4-a548-31e824e58070"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:34:13.473720 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:13.473699 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b559f06b-d917-4fb4-a548-31e824e58070-kube-api-access-2nfkm" (OuterVolumeSpecName: "kube-api-access-2nfkm") pod "b559f06b-d917-4fb4-a548-31e824e58070" (UID: "b559f06b-d917-4fb4-a548-31e824e58070"). InnerVolumeSpecName "kube-api-access-2nfkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:34:13.473788 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:13.473766 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b559f06b-d917-4fb4-a548-31e824e58070-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b559f06b-d917-4fb4-a548-31e824e58070" (UID: "b559f06b-d917-4fb4-a548-31e824e58070"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 20:34:13.573100 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:13.573069 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nfkm\" (UniqueName: \"kubernetes.io/projected/b559f06b-d917-4fb4-a548-31e824e58070-kube-api-access-2nfkm\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:34:13.573100 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:13.573095 2546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b559f06b-d917-4fb4-a548-31e824e58070-istio-token\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:34:13.573100 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:13.573104 2546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b559f06b-d917-4fb4-a548-31e824e58070-istio-podinfo\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:34:13.573295 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:13.573112 2546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b559f06b-d917-4fb4-a548-31e824e58070-istiod-ca-cert\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:34:13.573295 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:13.573122 2546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b559f06b-d917-4fb4-a548-31e824e58070-workload-certs\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:34:13.573295 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:13.573130 2546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b559f06b-d917-4fb4-a548-31e824e58070-workload-socket\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:34:13.573295 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:13.573139 2546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b559f06b-d917-4fb4-a548-31e824e58070-istio-envoy\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:34:13.771292 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:13.771262 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s"] Sep 11 20:34:13.775440 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:13.775414 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-fcb0e4-openshift-default-5d9d954488-mrw2s"] Sep 11 20:34:14.451418 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:14.451383 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b559f06b-d917-4fb4-a548-31e824e58070" path="/var/lib/kubelet/pods/b559f06b-d917-4fb4-a548-31e824e58070/volumes" Sep 11 20:34:18.926422 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:18.926387 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:34:19.007112 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:19.007081 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:34:19.068510 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:19.068471 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:34:19.130026 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:19.129990 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:34:19.181379 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:19.181299 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:34:19.478550 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:19.478441 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:34:19.577686 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:19.577652 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:34:25.172055 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:25.172016 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:34:25.284319 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:25.284278 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:34:25.436718 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:25.436642 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:34:27.943894 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:27.943860 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-57fc56fd56-pp52g"] Sep 11 20:34:27.944336 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:27.944177 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-57fc56fd56-pp52g" podUID="4798791b-9a9d-490f-9885-b3db73dfb831" containerName="manager" containerID="cri-o://42e336c5e20297a297926082a20b85521bf72025ed1171185378287ae16974f7" gracePeriod=30 Sep 11 20:34:28.184385 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:28.184364 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57fc56fd56-pp52g" Sep 11 20:34:28.311775 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:28.311700 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gtbzd\" (UniqueName: \"kubernetes.io/projected/4798791b-9a9d-490f-9885-b3db73dfb831-kube-api-access-gtbzd\") pod \"4798791b-9a9d-490f-9885-b3db73dfb831\" (UID: \"4798791b-9a9d-490f-9885-b3db73dfb831\") " Sep 11 20:34:28.311775 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:28.311744 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4798791b-9a9d-490f-9885-b3db73dfb831-webhook-cert\") pod \"4798791b-9a9d-490f-9885-b3db73dfb831\" (UID: \"4798791b-9a9d-490f-9885-b3db73dfb831\") " Sep 11 20:34:28.311954 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:28.311782 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4798791b-9a9d-490f-9885-b3db73dfb831-metrics-certs\") pod \"4798791b-9a9d-490f-9885-b3db73dfb831\" (UID: \"4798791b-9a9d-490f-9885-b3db73dfb831\") " Sep 11 20:34:28.313793 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:28.313756 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4798791b-9a9d-490f-9885-b3db73dfb831-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "4798791b-9a9d-490f-9885-b3db73dfb831" (UID: "4798791b-9a9d-490f-9885-b3db73dfb831"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 20:34:28.313793 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:28.313780 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4798791b-9a9d-490f-9885-b3db73dfb831-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4798791b-9a9d-490f-9885-b3db73dfb831" (UID: "4798791b-9a9d-490f-9885-b3db73dfb831"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 20:34:28.313958 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:28.313770 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4798791b-9a9d-490f-9885-b3db73dfb831-kube-api-access-gtbzd" (OuterVolumeSpecName: "kube-api-access-gtbzd") pod "4798791b-9a9d-490f-9885-b3db73dfb831" (UID: "4798791b-9a9d-490f-9885-b3db73dfb831"). InnerVolumeSpecName "kube-api-access-gtbzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:34:28.412890 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:28.412860 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtbzd\" (UniqueName: \"kubernetes.io/projected/4798791b-9a9d-490f-9885-b3db73dfb831-kube-api-access-gtbzd\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:34:28.412890 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:28.412890 2546 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4798791b-9a9d-490f-9885-b3db73dfb831-webhook-cert\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:34:28.413079 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:28.412903 2546 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4798791b-9a9d-490f-9885-b3db73dfb831-metrics-certs\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:34:28.530548 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:28.530518 2546 generic.go:356] "Generic (PLEG): container finished" podID="4798791b-9a9d-490f-9885-b3db73dfb831" containerID="42e336c5e20297a297926082a20b85521bf72025ed1171185378287ae16974f7" exitCode=0 Sep 11 20:34:28.530699 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:28.530567 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57fc56fd56-pp52g" Sep 11 20:34:28.530699 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:28.530600 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57fc56fd56-pp52g" event={"ID":"4798791b-9a9d-490f-9885-b3db73dfb831","Type":"ContainerDied","Data":"42e336c5e20297a297926082a20b85521bf72025ed1171185378287ae16974f7"} Sep 11 20:34:28.530699 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:28.530652 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57fc56fd56-pp52g" event={"ID":"4798791b-9a9d-490f-9885-b3db73dfb831","Type":"ContainerDied","Data":"c5f4a7eca758831a55217714e5f8d3e17676060f88911a8d0ed7540fe4fbe5d3"} Sep 11 20:34:28.530699 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:28.530681 2546 scope.go:117] "RemoveContainer" containerID="42e336c5e20297a297926082a20b85521bf72025ed1171185378287ae16974f7" Sep 11 20:34:28.540263 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:28.540235 2546 scope.go:117] "RemoveContainer" containerID="42e336c5e20297a297926082a20b85521bf72025ed1171185378287ae16974f7" Sep 11 20:34:28.540524 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:34:28.540504 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42e336c5e20297a297926082a20b85521bf72025ed1171185378287ae16974f7\": container with ID starting with 42e336c5e20297a297926082a20b85521bf72025ed1171185378287ae16974f7 not found: ID does not exist" containerID="42e336c5e20297a297926082a20b85521bf72025ed1171185378287ae16974f7" Sep 11 20:34:28.540590 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:28.540531 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42e336c5e20297a297926082a20b85521bf72025ed1171185378287ae16974f7"} err="failed to get container status \"42e336c5e20297a297926082a20b85521bf72025ed1171185378287ae16974f7\": rpc error: code = NotFound desc = could not find container \"42e336c5e20297a297926082a20b85521bf72025ed1171185378287ae16974f7\": container with ID starting with 42e336c5e20297a297926082a20b85521bf72025ed1171185378287ae16974f7 not found: ID does not exist" Sep 11 20:34:28.544731 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:28.544711 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-57fc56fd56-pp52g"] Sep 11 20:34:28.548341 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:28.548313 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-57fc56fd56-pp52g"] Sep 11 20:34:30.452981 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:34:30.452943 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4798791b-9a9d-490f-9885-b3db73dfb831" path="/var/lib/kubelet/pods/4798791b-9a9d-490f-9885-b3db73dfb831/volumes" Sep 11 20:35:29.315160 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:29.315123 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-57fc56fd56-vg9lm"] Sep 11 20:35:29.315672 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:29.315554 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b559f06b-d917-4fb4-a548-31e824e58070" containerName="istio-proxy" Sep 11 20:35:29.315672 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:29.315567 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="b559f06b-d917-4fb4-a548-31e824e58070" containerName="istio-proxy" Sep 11 20:35:29.315672 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:29.315601 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3794824d-d5d9-4f3a-bcda-75e80942a10a" containerName="payload-pre-processing" Sep 11 20:35:29.315672 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:29.315608 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="3794824d-d5d9-4f3a-bcda-75e80942a10a" containerName="payload-pre-processing" Sep 11 20:35:29.315672 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:29.315621 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="461ac6f0-a3e8-4526-849a-290a9872e785" containerName="payload-processing" Sep 11 20:35:29.315672 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:29.315628 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="461ac6f0-a3e8-4526-849a-290a9872e785" containerName="payload-processing" Sep 11 20:35:29.315672 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:29.315634 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0675d2f-779a-4e4d-8669-fa774841dab4" containerName="istio-proxy" Sep 11 20:35:29.315672 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:29.315638 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0675d2f-779a-4e4d-8669-fa774841dab4" containerName="istio-proxy" Sep 11 20:35:29.315672 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:29.315646 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4798791b-9a9d-490f-9885-b3db73dfb831" containerName="manager" Sep 11 20:35:29.315672 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:29.315652 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="4798791b-9a9d-490f-9885-b3db73dfb831" containerName="manager" Sep 11 20:35:29.315672 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:29.315668 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e34b67c7-d6e4-4819-a353-76004387dc4b" containerName="payload-pre-processing" Sep 11 20:35:29.315672 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:29.315676 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="e34b67c7-d6e4-4819-a353-76004387dc4b" containerName="payload-pre-processing" Sep 11 20:35:29.316043 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:29.315690 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6477778-0da8-45d2-bc2f-222e7f7a9d2f" containerName="maas-api" Sep 11 20:35:29.316043 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:29.315696 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6477778-0da8-45d2-bc2f-222e7f7a9d2f" containerName="maas-api" Sep 11 20:35:29.316043 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:29.315705 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cc670c4-aa51-441e-9027-68c0886aab39" containerName="payload-pre-processing" Sep 11 20:35:29.316043 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:29.315711 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cc670c4-aa51-441e-9027-68c0886aab39" containerName="payload-pre-processing" Sep 11 20:35:29.316043 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:29.315717 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdb1c9ca-d879-4d6f-b470-022dae6a45cf" containerName="payload-processing" Sep 11 20:35:29.316043 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:29.315722 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdb1c9ca-d879-4d6f-b470-022dae6a45cf" containerName="payload-processing" Sep 11 20:35:29.316043 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:29.315780 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="3794824d-d5d9-4f3a-bcda-75e80942a10a" containerName="payload-pre-processing" Sep 11 20:35:29.316043 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:29.315788 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="461ac6f0-a3e8-4526-849a-290a9872e785" containerName="payload-processing" Sep 11 20:35:29.316043 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:29.315794 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0675d2f-779a-4e4d-8669-fa774841dab4" containerName="istio-proxy" Sep 11 20:35:29.316043 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:29.315799 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="e34b67c7-d6e4-4819-a353-76004387dc4b" containerName="payload-pre-processing" Sep 11 20:35:29.316043 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:29.315805 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6477778-0da8-45d2-bc2f-222e7f7a9d2f" containerName="maas-api" Sep 11 20:35:29.316043 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:29.315810 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="4798791b-9a9d-490f-9885-b3db73dfb831" containerName="manager" Sep 11 20:35:29.316043 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:29.315815 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdb1c9ca-d879-4d6f-b470-022dae6a45cf" containerName="payload-processing" Sep 11 20:35:29.316043 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:29.315823 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cc670c4-aa51-441e-9027-68c0886aab39" containerName="payload-pre-processing" Sep 11 20:35:29.316043 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:29.315832 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="b559f06b-d917-4fb4-a548-31e824e58070" containerName="istio-proxy" Sep 11 20:35:29.316043 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:29.315839 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="b559f06b-d917-4fb4-a548-31e824e58070" containerName="istio-proxy" Sep 11 20:35:29.320173 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:29.320130 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57fc56fd56-vg9lm" Sep 11 20:35:29.324086 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:29.323256 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-2grdg\"" Sep 11 20:35:29.324086 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:29.323517 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 11 20:35:29.324086 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:29.323759 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 11 20:35:29.324086 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:29.323979 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 11 20:35:29.327497 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:29.327475 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57fc56fd56-vg9lm"] Sep 11 20:35:29.466773 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:29.466732 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcqzw\" (UniqueName: \"kubernetes.io/projected/ab4405f1-728c-4751-876f-c84049292128-kube-api-access-zcqzw\") pod \"maas-controller-57fc56fd56-vg9lm\" (UID: \"ab4405f1-728c-4751-876f-c84049292128\") " pod="opendatahub/maas-controller-57fc56fd56-vg9lm" Sep 11 20:35:29.466986 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:29.466900 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab4405f1-728c-4751-876f-c84049292128-metrics-certs\") pod \"maas-controller-57fc56fd56-vg9lm\" (UID: \"ab4405f1-728c-4751-876f-c84049292128\") " pod="opendatahub/maas-controller-57fc56fd56-vg9lm" Sep 11 20:35:29.466986 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:29.466951 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ab4405f1-728c-4751-876f-c84049292128-webhook-cert\") pod \"maas-controller-57fc56fd56-vg9lm\" (UID: \"ab4405f1-728c-4751-876f-c84049292128\") " pod="opendatahub/maas-controller-57fc56fd56-vg9lm" Sep 11 20:35:29.567688 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:29.567569 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcqzw\" (UniqueName: \"kubernetes.io/projected/ab4405f1-728c-4751-876f-c84049292128-kube-api-access-zcqzw\") pod \"maas-controller-57fc56fd56-vg9lm\" (UID: \"ab4405f1-728c-4751-876f-c84049292128\") " pod="opendatahub/maas-controller-57fc56fd56-vg9lm" Sep 11 20:35:29.567851 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:29.567688 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab4405f1-728c-4751-876f-c84049292128-metrics-certs\") pod \"maas-controller-57fc56fd56-vg9lm\" (UID: \"ab4405f1-728c-4751-876f-c84049292128\") " pod="opendatahub/maas-controller-57fc56fd56-vg9lm" Sep 11 20:35:29.567851 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:29.567715 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ab4405f1-728c-4751-876f-c84049292128-webhook-cert\") pod \"maas-controller-57fc56fd56-vg9lm\" (UID: \"ab4405f1-728c-4751-876f-c84049292128\") " pod="opendatahub/maas-controller-57fc56fd56-vg9lm" Sep 11 20:35:29.570066 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:29.570036 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ab4405f1-728c-4751-876f-c84049292128-webhook-cert\") pod \"maas-controller-57fc56fd56-vg9lm\" (UID: \"ab4405f1-728c-4751-876f-c84049292128\") " pod="opendatahub/maas-controller-57fc56fd56-vg9lm" Sep 11 20:35:29.570187 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:29.570101 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab4405f1-728c-4751-876f-c84049292128-metrics-certs\") pod \"maas-controller-57fc56fd56-vg9lm\" (UID: \"ab4405f1-728c-4751-876f-c84049292128\") " pod="opendatahub/maas-controller-57fc56fd56-vg9lm" Sep 11 20:35:29.575336 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:29.575315 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcqzw\" (UniqueName: \"kubernetes.io/projected/ab4405f1-728c-4751-876f-c84049292128-kube-api-access-zcqzw\") pod \"maas-controller-57fc56fd56-vg9lm\" (UID: \"ab4405f1-728c-4751-876f-c84049292128\") " pod="opendatahub/maas-controller-57fc56fd56-vg9lm" Sep 11 20:35:29.634199 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:29.634172 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57fc56fd56-vg9lm" Sep 11 20:35:29.902651 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:29.902422 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57fc56fd56-vg9lm"] Sep 11 20:35:29.905414 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:35:29.905381 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab4405f1_728c_4751_876f_c84049292128.slice/crio-391f3ed7e57fa3b3a1c983bdb684feeded5838a7e95be0a3977821605e26f0aa WatchSource:0}: Error finding container 391f3ed7e57fa3b3a1c983bdb684feeded5838a7e95be0a3977821605e26f0aa: Status 404 returned error can't find the container with id 391f3ed7e57fa3b3a1c983bdb684feeded5838a7e95be0a3977821605e26f0aa Sep 11 20:35:29.906827 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:29.906807 2546 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 20:35:30.385663 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:30.385638 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 11 20:35:30.849361 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:30.849322 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57fc56fd56-vg9lm" event={"ID":"ab4405f1-728c-4751-876f-c84049292128","Type":"ContainerStarted","Data":"fc64fff90d042d4e4b6f968cf8124b335406d8dd93dd894cab9699fe7388bd75"} Sep 11 20:35:30.849533 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:30.849388 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57fc56fd56-vg9lm" event={"ID":"ab4405f1-728c-4751-876f-c84049292128","Type":"ContainerStarted","Data":"391f3ed7e57fa3b3a1c983bdb684feeded5838a7e95be0a3977821605e26f0aa"} Sep 11 20:35:30.849533 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:30.849420 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-57fc56fd56-vg9lm" Sep 11 20:35:30.863620 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:30.863550 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-57fc56fd56-vg9lm" podStartSLOduration=1.386681968 podStartE2EDuration="1.863529779s" totalImagesPullingTime="476.847811ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:35:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:35:30.863030172 +0000 UTC m=+2100.911080543" watchObservedRunningTime="2026-09-11 20:35:30.863529779 +0000 UTC m=+2100.911580123" Sep 11 20:35:41.857733 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:41.857697 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-57fc56fd56-vg9lm" Sep 11 20:35:51.073136 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:51.073106 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:35:51.160953 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:51.160918 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-57fc56fd56-vg9lm"] Sep 11 20:35:51.161155 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:51.161129 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-57fc56fd56-vg9lm" podUID="ab4405f1-728c-4751-876f-c84049292128" containerName="manager" containerID="cri-o://fc64fff90d042d4e4b6f968cf8124b335406d8dd93dd894cab9699fe7388bd75" gracePeriod=30 Sep 11 20:35:51.404848 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:51.404825 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57fc56fd56-vg9lm" Sep 11 20:35:51.472999 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:51.472972 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ab4405f1-728c-4751-876f-c84049292128-webhook-cert\") pod \"ab4405f1-728c-4751-876f-c84049292128\" (UID: \"ab4405f1-728c-4751-876f-c84049292128\") " Sep 11 20:35:51.473177 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:51.473004 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zcqzw\" (UniqueName: \"kubernetes.io/projected/ab4405f1-728c-4751-876f-c84049292128-kube-api-access-zcqzw\") pod \"ab4405f1-728c-4751-876f-c84049292128\" (UID: \"ab4405f1-728c-4751-876f-c84049292128\") " Sep 11 20:35:51.473177 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:51.473051 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab4405f1-728c-4751-876f-c84049292128-metrics-certs\") pod \"ab4405f1-728c-4751-876f-c84049292128\" (UID: \"ab4405f1-728c-4751-876f-c84049292128\") " Sep 11 20:35:51.475078 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:51.475051 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab4405f1-728c-4751-876f-c84049292128-kube-api-access-zcqzw" (OuterVolumeSpecName: "kube-api-access-zcqzw") pod "ab4405f1-728c-4751-876f-c84049292128" (UID: "ab4405f1-728c-4751-876f-c84049292128"). InnerVolumeSpecName "kube-api-access-zcqzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:35:51.475193 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:51.475115 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab4405f1-728c-4751-876f-c84049292128-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "ab4405f1-728c-4751-876f-c84049292128" (UID: "ab4405f1-728c-4751-876f-c84049292128"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 20:35:51.475267 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:51.475246 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab4405f1-728c-4751-876f-c84049292128-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ab4405f1-728c-4751-876f-c84049292128" (UID: "ab4405f1-728c-4751-876f-c84049292128"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 20:35:51.573908 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:51.573875 2546 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ab4405f1-728c-4751-876f-c84049292128-webhook-cert\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:35:51.573908 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:51.573900 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcqzw\" (UniqueName: \"kubernetes.io/projected/ab4405f1-728c-4751-876f-c84049292128-kube-api-access-zcqzw\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:35:51.573908 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:51.573911 2546 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab4405f1-728c-4751-876f-c84049292128-metrics-certs\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:35:51.974921 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:51.974889 2546 generic.go:356] "Generic (PLEG): container finished" podID="ab4405f1-728c-4751-876f-c84049292128" containerID="fc64fff90d042d4e4b6f968cf8124b335406d8dd93dd894cab9699fe7388bd75" exitCode=0 Sep 11 20:35:51.975080 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:51.974934 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57fc56fd56-vg9lm" Sep 11 20:35:51.975080 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:51.974969 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57fc56fd56-vg9lm" event={"ID":"ab4405f1-728c-4751-876f-c84049292128","Type":"ContainerDied","Data":"fc64fff90d042d4e4b6f968cf8124b335406d8dd93dd894cab9699fe7388bd75"} Sep 11 20:35:51.975080 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:51.975022 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57fc56fd56-vg9lm" event={"ID":"ab4405f1-728c-4751-876f-c84049292128","Type":"ContainerDied","Data":"391f3ed7e57fa3b3a1c983bdb684feeded5838a7e95be0a3977821605e26f0aa"} Sep 11 20:35:51.975080 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:51.975046 2546 scope.go:117] "RemoveContainer" containerID="fc64fff90d042d4e4b6f968cf8124b335406d8dd93dd894cab9699fe7388bd75" Sep 11 20:35:51.984699 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:51.984684 2546 scope.go:117] "RemoveContainer" containerID="fc64fff90d042d4e4b6f968cf8124b335406d8dd93dd894cab9699fe7388bd75" Sep 11 20:35:51.984980 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:35:51.984958 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc64fff90d042d4e4b6f968cf8124b335406d8dd93dd894cab9699fe7388bd75\": container with ID starting with fc64fff90d042d4e4b6f968cf8124b335406d8dd93dd894cab9699fe7388bd75 not found: ID does not exist" containerID="fc64fff90d042d4e4b6f968cf8124b335406d8dd93dd894cab9699fe7388bd75" Sep 11 20:35:51.985053 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:51.984988 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc64fff90d042d4e4b6f968cf8124b335406d8dd93dd894cab9699fe7388bd75"} err="failed to get container status \"fc64fff90d042d4e4b6f968cf8124b335406d8dd93dd894cab9699fe7388bd75\": rpc error: code = NotFound desc = could not find container \"fc64fff90d042d4e4b6f968cf8124b335406d8dd93dd894cab9699fe7388bd75\": container with ID starting with fc64fff90d042d4e4b6f968cf8124b335406d8dd93dd894cab9699fe7388bd75 not found: ID does not exist" Sep 11 20:35:51.995635 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:51.995613 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-57fc56fd56-vg9lm"] Sep 11 20:35:51.999040 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:51.999020 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-57fc56fd56-vg9lm"] Sep 11 20:35:52.453095 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:52.453052 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab4405f1-728c-4751-876f-c84049292128" path="/var/lib/kubelet/pods/ab4405f1-728c-4751-876f-c84049292128/volumes" Sep 11 20:35:52.748204 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:52.748113 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-57fc56fd56-srkrt"] Sep 11 20:35:52.748649 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:52.748623 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab4405f1-728c-4751-876f-c84049292128" containerName="manager" Sep 11 20:35:52.748649 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:52.748647 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab4405f1-728c-4751-876f-c84049292128" containerName="manager" Sep 11 20:35:52.748872 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:52.748668 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b559f06b-d917-4fb4-a548-31e824e58070" containerName="istio-proxy" Sep 11 20:35:52.748872 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:52.748677 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="b559f06b-d917-4fb4-a548-31e824e58070" containerName="istio-proxy" Sep 11 20:35:52.748872 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:52.748693 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0675d2f-779a-4e4d-8669-fa774841dab4" containerName="istio-proxy" Sep 11 20:35:52.748872 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:52.748701 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0675d2f-779a-4e4d-8669-fa774841dab4" containerName="istio-proxy" Sep 11 20:35:52.748872 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:52.748821 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0675d2f-779a-4e4d-8669-fa774841dab4" containerName="istio-proxy" Sep 11 20:35:52.748872 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:52.748834 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab4405f1-728c-4751-876f-c84049292128" containerName="manager" Sep 11 20:35:52.753467 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:52.753447 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57fc56fd56-srkrt" Sep 11 20:35:52.755639 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:52.755618 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 11 20:35:52.756539 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:52.756475 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-2grdg\"" Sep 11 20:35:52.756539 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:52.756488 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 11 20:35:52.756896 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:52.756873 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 11 20:35:52.759196 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:52.759175 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57fc56fd56-srkrt"] Sep 11 20:35:52.885018 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:52.884987 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trs27\" (UniqueName: \"kubernetes.io/projected/27ecd431-11ab-4ad3-9f75-4d79e7de99f7-kube-api-access-trs27\") pod \"maas-controller-57fc56fd56-srkrt\" (UID: \"27ecd431-11ab-4ad3-9f75-4d79e7de99f7\") " pod="opendatahub/maas-controller-57fc56fd56-srkrt" Sep 11 20:35:52.885162 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:52.885045 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/27ecd431-11ab-4ad3-9f75-4d79e7de99f7-webhook-cert\") pod \"maas-controller-57fc56fd56-srkrt\" (UID: \"27ecd431-11ab-4ad3-9f75-4d79e7de99f7\") " pod="opendatahub/maas-controller-57fc56fd56-srkrt" Sep 11 20:35:52.885162 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:52.885132 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27ecd431-11ab-4ad3-9f75-4d79e7de99f7-metrics-certs\") pod \"maas-controller-57fc56fd56-srkrt\" (UID: \"27ecd431-11ab-4ad3-9f75-4d79e7de99f7\") " pod="opendatahub/maas-controller-57fc56fd56-srkrt" Sep 11 20:35:52.985946 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:52.985918 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trs27\" (UniqueName: \"kubernetes.io/projected/27ecd431-11ab-4ad3-9f75-4d79e7de99f7-kube-api-access-trs27\") pod \"maas-controller-57fc56fd56-srkrt\" (UID: \"27ecd431-11ab-4ad3-9f75-4d79e7de99f7\") " pod="opendatahub/maas-controller-57fc56fd56-srkrt" Sep 11 20:35:52.986120 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:52.985970 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/27ecd431-11ab-4ad3-9f75-4d79e7de99f7-webhook-cert\") pod \"maas-controller-57fc56fd56-srkrt\" (UID: \"27ecd431-11ab-4ad3-9f75-4d79e7de99f7\") " pod="opendatahub/maas-controller-57fc56fd56-srkrt" Sep 11 20:35:52.986120 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:52.986034 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27ecd431-11ab-4ad3-9f75-4d79e7de99f7-metrics-certs\") pod \"maas-controller-57fc56fd56-srkrt\" (UID: \"27ecd431-11ab-4ad3-9f75-4d79e7de99f7\") " pod="opendatahub/maas-controller-57fc56fd56-srkrt" Sep 11 20:35:52.988387 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:52.988364 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/27ecd431-11ab-4ad3-9f75-4d79e7de99f7-webhook-cert\") pod \"maas-controller-57fc56fd56-srkrt\" (UID: \"27ecd431-11ab-4ad3-9f75-4d79e7de99f7\") " pod="opendatahub/maas-controller-57fc56fd56-srkrt" Sep 11 20:35:52.988489 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:52.988471 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27ecd431-11ab-4ad3-9f75-4d79e7de99f7-metrics-certs\") pod \"maas-controller-57fc56fd56-srkrt\" (UID: \"27ecd431-11ab-4ad3-9f75-4d79e7de99f7\") " pod="opendatahub/maas-controller-57fc56fd56-srkrt" Sep 11 20:35:52.993502 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:52.993482 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trs27\" (UniqueName: \"kubernetes.io/projected/27ecd431-11ab-4ad3-9f75-4d79e7de99f7-kube-api-access-trs27\") pod \"maas-controller-57fc56fd56-srkrt\" (UID: \"27ecd431-11ab-4ad3-9f75-4d79e7de99f7\") " pod="opendatahub/maas-controller-57fc56fd56-srkrt" Sep 11 20:35:53.065617 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:53.065531 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57fc56fd56-srkrt" Sep 11 20:35:53.338117 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:53.338094 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57fc56fd56-srkrt"] Sep 11 20:35:53.340244 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:35:53.340218 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27ecd431_11ab_4ad3_9f75_4d79e7de99f7.slice/crio-eb389980752c51795d175cb1f91a87b3c8b911351431044a603d43e4fa404e2b WatchSource:0}: Error finding container eb389980752c51795d175cb1f91a87b3c8b911351431044a603d43e4fa404e2b: Status 404 returned error can't find the container with id eb389980752c51795d175cb1f91a87b3c8b911351431044a603d43e4fa404e2b Sep 11 20:35:53.370813 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:53.370787 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:35:53.988551 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:53.988450 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57fc56fd56-srkrt" event={"ID":"27ecd431-11ab-4ad3-9f75-4d79e7de99f7","Type":"ContainerStarted","Data":"41974d8c6ed2f8a0b9fc2ee328c70a73080119643a69a298a221b64bc96f76b1"} Sep 11 20:35:53.988551 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:53.988547 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57fc56fd56-srkrt" event={"ID":"27ecd431-11ab-4ad3-9f75-4d79e7de99f7","Type":"ContainerStarted","Data":"eb389980752c51795d175cb1f91a87b3c8b911351431044a603d43e4fa404e2b"} Sep 11 20:35:53.989048 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:53.988698 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-57fc56fd56-srkrt" Sep 11 20:35:54.009206 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:35:54.009164 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-57fc56fd56-srkrt" podStartSLOduration=1.61776307 podStartE2EDuration="2.009149087s" totalImagesPullingTime="391.386017ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:35:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:35:54.008123309 +0000 UTC m=+2124.056173653" watchObservedRunningTime="2026-09-11 20:35:54.009149087 +0000 UTC m=+2124.057199431" Sep 11 20:36:04.996658 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:36:04.996606 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-57fc56fd56-srkrt" Sep 11 20:38:08.711818 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:38:08.711785 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:38:15.754600 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:38:15.754552 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:38:15.808946 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:38:15.808923 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:38:43.749027 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:38:43.748991 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:38:47.131568 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:38:47.131512 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:38:49.308529 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:38:49.308497 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:39:01.207531 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:39:01.207474 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:39:27.201628 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:39:27.201536 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:39:28.682141 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:39:28.682108 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:39:54.411046 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:39:54.411014 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c564f7cc4-f5lb9"] Sep 11 20:39:54.414568 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:39:54.414548 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c564f7cc4-f5lb9" Sep 11 20:39:54.423457 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:39:54.423433 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c564f7cc4-f5lb9"] Sep 11 20:39:54.520042 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:39:54.520012 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfdhd\" (UniqueName: \"kubernetes.io/projected/56d22a0c-06e1-45b3-a192-ae704c5c4bb1-kube-api-access-vfdhd\") pod \"maas-api-5c564f7cc4-f5lb9\" (UID: \"56d22a0c-06e1-45b3-a192-ae704c5c4bb1\") " pod="odh-ai-gateway-infra/maas-api-5c564f7cc4-f5lb9" Sep 11 20:39:54.520205 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:39:54.520070 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56d22a0c-06e1-45b3-a192-ae704c5c4bb1-openshift-service-ca\") pod \"maas-api-5c564f7cc4-f5lb9\" (UID: \"56d22a0c-06e1-45b3-a192-ae704c5c4bb1\") " pod="odh-ai-gateway-infra/maas-api-5c564f7cc4-f5lb9" Sep 11 20:39:54.520205 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:39:54.520116 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56d22a0c-06e1-45b3-a192-ae704c5c4bb1-maas-api-tls\") pod \"maas-api-5c564f7cc4-f5lb9\" (UID: \"56d22a0c-06e1-45b3-a192-ae704c5c4bb1\") " pod="odh-ai-gateway-infra/maas-api-5c564f7cc4-f5lb9" Sep 11 20:39:54.520205 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:39:54.520185 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56d22a0c-06e1-45b3-a192-ae704c5c4bb1-metrics-certs\") pod \"maas-api-5c564f7cc4-f5lb9\" (UID: \"56d22a0c-06e1-45b3-a192-ae704c5c4bb1\") " pod="odh-ai-gateway-infra/maas-api-5c564f7cc4-f5lb9" Sep 11 20:39:54.621288 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:39:54.621257 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56d22a0c-06e1-45b3-a192-ae704c5c4bb1-openshift-service-ca\") pod \"maas-api-5c564f7cc4-f5lb9\" (UID: \"56d22a0c-06e1-45b3-a192-ae704c5c4bb1\") " pod="odh-ai-gateway-infra/maas-api-5c564f7cc4-f5lb9" Sep 11 20:39:54.621439 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:39:54.621307 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56d22a0c-06e1-45b3-a192-ae704c5c4bb1-maas-api-tls\") pod \"maas-api-5c564f7cc4-f5lb9\" (UID: \"56d22a0c-06e1-45b3-a192-ae704c5c4bb1\") " pod="odh-ai-gateway-infra/maas-api-5c564f7cc4-f5lb9" Sep 11 20:39:54.621510 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:39:54.621483 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56d22a0c-06e1-45b3-a192-ae704c5c4bb1-metrics-certs\") pod \"maas-api-5c564f7cc4-f5lb9\" (UID: \"56d22a0c-06e1-45b3-a192-ae704c5c4bb1\") " pod="odh-ai-gateway-infra/maas-api-5c564f7cc4-f5lb9" Sep 11 20:39:54.621673 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:39:54.621656 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfdhd\" (UniqueName: \"kubernetes.io/projected/56d22a0c-06e1-45b3-a192-ae704c5c4bb1-kube-api-access-vfdhd\") pod \"maas-api-5c564f7cc4-f5lb9\" (UID: \"56d22a0c-06e1-45b3-a192-ae704c5c4bb1\") " pod="odh-ai-gateway-infra/maas-api-5c564f7cc4-f5lb9" Sep 11 20:39:54.622073 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:39:54.622042 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56d22a0c-06e1-45b3-a192-ae704c5c4bb1-openshift-service-ca\") pod \"maas-api-5c564f7cc4-f5lb9\" (UID: \"56d22a0c-06e1-45b3-a192-ae704c5c4bb1\") " pod="odh-ai-gateway-infra/maas-api-5c564f7cc4-f5lb9" Sep 11 20:39:54.623903 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:39:54.623880 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56d22a0c-06e1-45b3-a192-ae704c5c4bb1-metrics-certs\") pod \"maas-api-5c564f7cc4-f5lb9\" (UID: \"56d22a0c-06e1-45b3-a192-ae704c5c4bb1\") " pod="odh-ai-gateway-infra/maas-api-5c564f7cc4-f5lb9" Sep 11 20:39:54.623967 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:39:54.623914 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56d22a0c-06e1-45b3-a192-ae704c5c4bb1-maas-api-tls\") pod \"maas-api-5c564f7cc4-f5lb9\" (UID: \"56d22a0c-06e1-45b3-a192-ae704c5c4bb1\") " pod="odh-ai-gateway-infra/maas-api-5c564f7cc4-f5lb9" Sep 11 20:39:54.628551 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:39:54.628532 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfdhd\" (UniqueName: \"kubernetes.io/projected/56d22a0c-06e1-45b3-a192-ae704c5c4bb1-kube-api-access-vfdhd\") pod \"maas-api-5c564f7cc4-f5lb9\" (UID: \"56d22a0c-06e1-45b3-a192-ae704c5c4bb1\") " pod="odh-ai-gateway-infra/maas-api-5c564f7cc4-f5lb9" Sep 11 20:39:54.726540 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:39:54.726460 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c564f7cc4-f5lb9" Sep 11 20:39:55.000405 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:39:55.000370 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c564f7cc4-f5lb9"] Sep 11 20:39:55.002725 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:39:55.002693 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56d22a0c_06e1_45b3_a192_ae704c5c4bb1.slice/crio-be59c20d0b005635a4fb74ede28af6318be738b4f56d75e82c0f6f0122a234e2 WatchSource:0}: Error finding container be59c20d0b005635a4fb74ede28af6318be738b4f56d75e82c0f6f0122a234e2: Status 404 returned error can't find the container with id be59c20d0b005635a4fb74ede28af6318be738b4f56d75e82c0f6f0122a234e2 Sep 11 20:39:55.188929 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:39:55.188890 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c564f7cc4-f5lb9" event={"ID":"56d22a0c-06e1-45b3-a192-ae704c5c4bb1","Type":"ContainerStarted","Data":"be59c20d0b005635a4fb74ede28af6318be738b4f56d75e82c0f6f0122a234e2"} Sep 11 20:39:56.195542 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:39:56.195497 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c564f7cc4-f5lb9" event={"ID":"56d22a0c-06e1-45b3-a192-ae704c5c4bb1","Type":"ContainerStarted","Data":"3183aeac0683116a41adbd6001fe321a59b330912d90ac579af5b2ab3e31b8d0"} Sep 11 20:39:56.196176 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:39:56.196147 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c564f7cc4-f5lb9" Sep 11 20:39:56.211152 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:39:56.211099 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c564f7cc4-f5lb9" podStartSLOduration=1.895895285 podStartE2EDuration="2.211083572s" totalImagesPullingTime="315.188287ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:39:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:39:56.210116409 +0000 UTC m=+2366.258166792" watchObservedRunningTime="2026-09-11 20:39:56.211083572 +0000 UTC m=+2366.259133940" Sep 11 20:40:00.133729 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:00.133699 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:40:02.205492 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:02.205464 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c564f7cc4-f5lb9" Sep 11 20:40:02.251398 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:02.251363 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-85cff974b6-78c9p"] Sep 11 20:40:02.251716 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:02.251686 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-85cff974b6-78c9p" podUID="62df1741-005c-49a4-8166-faf72dd73a39" containerName="maas-api" containerID="cri-o://d53c179acfc4825d0a6ec283986a1f99c69cd8c9de1889564a09d71881d6777a" gracePeriod=30 Sep 11 20:40:03.178327 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:03.178277 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-85cff974b6-78c9p" podUID="62df1741-005c-49a4-8166-faf72dd73a39" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.39:8443/health\": dial tcp 10.133.0.39:8443: connect: connection refused" Sep 11 20:40:03.500698 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:03.500674 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-85cff974b6-78c9p" Sep 11 20:40:03.504433 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:03.504411 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8222\" (UniqueName: \"kubernetes.io/projected/62df1741-005c-49a4-8166-faf72dd73a39-kube-api-access-n8222\") pod \"62df1741-005c-49a4-8166-faf72dd73a39\" (UID: \"62df1741-005c-49a4-8166-faf72dd73a39\") " Sep 11 20:40:03.504562 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:03.504485 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62df1741-005c-49a4-8166-faf72dd73a39-metrics-certs\") pod \"62df1741-005c-49a4-8166-faf72dd73a39\" (UID: \"62df1741-005c-49a4-8166-faf72dd73a39\") " Sep 11 20:40:03.504629 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:03.504590 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62df1741-005c-49a4-8166-faf72dd73a39-openshift-service-ca\") pod \"62df1741-005c-49a4-8166-faf72dd73a39\" (UID: \"62df1741-005c-49a4-8166-faf72dd73a39\") " Sep 11 20:40:03.504672 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:03.504631 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62df1741-005c-49a4-8166-faf72dd73a39-maas-api-tls\") pod \"62df1741-005c-49a4-8166-faf72dd73a39\" (UID: \"62df1741-005c-49a4-8166-faf72dd73a39\") " Sep 11 20:40:03.504938 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:03.504907 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62df1741-005c-49a4-8166-faf72dd73a39-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "62df1741-005c-49a4-8166-faf72dd73a39" (UID: "62df1741-005c-49a4-8166-faf72dd73a39"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:40:03.505228 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:03.505203 2546 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62df1741-005c-49a4-8166-faf72dd73a39-openshift-service-ca\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:40:03.506778 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:03.506745 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62df1741-005c-49a4-8166-faf72dd73a39-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "62df1741-005c-49a4-8166-faf72dd73a39" (UID: "62df1741-005c-49a4-8166-faf72dd73a39"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 20:40:03.506921 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:03.506904 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62df1741-005c-49a4-8166-faf72dd73a39-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "62df1741-005c-49a4-8166-faf72dd73a39" (UID: "62df1741-005c-49a4-8166-faf72dd73a39"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 20:40:03.507024 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:03.507003 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62df1741-005c-49a4-8166-faf72dd73a39-kube-api-access-n8222" (OuterVolumeSpecName: "kube-api-access-n8222") pod "62df1741-005c-49a4-8166-faf72dd73a39" (UID: "62df1741-005c-49a4-8166-faf72dd73a39"). InnerVolumeSpecName "kube-api-access-n8222". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:40:03.606024 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:03.605993 2546 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62df1741-005c-49a4-8166-faf72dd73a39-metrics-certs\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:40:03.606024 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:03.606020 2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62df1741-005c-49a4-8166-faf72dd73a39-maas-api-tls\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:40:03.606024 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:03.606029 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8222\" (UniqueName: \"kubernetes.io/projected/62df1741-005c-49a4-8166-faf72dd73a39-kube-api-access-n8222\") on node \"ip-10-0-138-205.ec2.internal\" DevicePath \"\"" Sep 11 20:40:04.242819 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:04.242781 2546 generic.go:356] "Generic (PLEG): container finished" podID="62df1741-005c-49a4-8166-faf72dd73a39" containerID="d53c179acfc4825d0a6ec283986a1f99c69cd8c9de1889564a09d71881d6777a" exitCode=0 Sep 11 20:40:04.243031 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:04.242832 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-85cff974b6-78c9p" Sep 11 20:40:04.243031 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:04.242872 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-85cff974b6-78c9p" event={"ID":"62df1741-005c-49a4-8166-faf72dd73a39","Type":"ContainerDied","Data":"d53c179acfc4825d0a6ec283986a1f99c69cd8c9de1889564a09d71881d6777a"} Sep 11 20:40:04.243031 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:04.242935 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-85cff974b6-78c9p" event={"ID":"62df1741-005c-49a4-8166-faf72dd73a39","Type":"ContainerDied","Data":"860a1aa092ed3bae475e905b340635e22d8049e1290c5d49cb679dca47fa1bf7"} Sep 11 20:40:04.243031 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:04.242962 2546 scope.go:117] "RemoveContainer" containerID="d53c179acfc4825d0a6ec283986a1f99c69cd8c9de1889564a09d71881d6777a" Sep 11 20:40:04.252326 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:04.252309 2546 scope.go:117] "RemoveContainer" containerID="d53c179acfc4825d0a6ec283986a1f99c69cd8c9de1889564a09d71881d6777a" Sep 11 20:40:04.252586 ip-10-0-138-205 kubenswrapper[2546]: E0911 20:40:04.252555 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d53c179acfc4825d0a6ec283986a1f99c69cd8c9de1889564a09d71881d6777a\": container with ID starting with d53c179acfc4825d0a6ec283986a1f99c69cd8c9de1889564a09d71881d6777a not found: ID does not exist" containerID="d53c179acfc4825d0a6ec283986a1f99c69cd8c9de1889564a09d71881d6777a" Sep 11 20:40:04.252640 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:04.252601 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d53c179acfc4825d0a6ec283986a1f99c69cd8c9de1889564a09d71881d6777a"} err="failed to get container status \"d53c179acfc4825d0a6ec283986a1f99c69cd8c9de1889564a09d71881d6777a\": rpc error: code = NotFound desc = could not find container \"d53c179acfc4825d0a6ec283986a1f99c69cd8c9de1889564a09d71881d6777a\": container with ID starting with d53c179acfc4825d0a6ec283986a1f99c69cd8c9de1889564a09d71881d6777a not found: ID does not exist" Sep 11 20:40:04.263326 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:04.263302 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-85cff974b6-78c9p"] Sep 11 20:40:04.267119 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:04.267097 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-85cff974b6-78c9p"] Sep 11 20:40:04.450521 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:04.450493 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62df1741-005c-49a4-8166-faf72dd73a39" path="/var/lib/kubelet/pods/62df1741-005c-49a4-8166-faf72dd73a39/volumes" Sep 11 20:40:21.673954 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:21.673918 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-xzdbn_7c00411d-ee5f-4b0f-8c45-2daa2426389d/manager/0.log" Sep 11 20:40:21.788849 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:21.788811 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-hs68x_9c8afe06-4c3c-41dd-9adb-f3cdc51ee68d/manager/0.log" Sep 11 20:40:21.907636 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:21.907606 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-57fc56fd56-srkrt_27ecd431-11ab-4ad3-9f75-4d79e7de99f7/manager/0.log" Sep 11 20:40:22.019786 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:22.019709 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-hmfh4_30b6578d-630f-4290-ad87-b335c2eef845/server/0.log" Sep 11 20:40:22.131592 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:22.131559 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-htbhw_0c85ece7-fbaf-4a92-99d5-e48a0170cd2d/manager/0.log" Sep 11 20:40:22.353460 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:22.353430 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5fdf48ff49-xthdd_be81dfc2-533a-441c-950c-83a3c63a4c6c/copy-manifests/0.log" Sep 11 20:40:22.364642 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:22.364615 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5fdf48ff49-xthdd_be81dfc2-533a-441c-950c-83a3c63a4c6c/manager/0.log" Sep 11 20:40:22.942161 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:22.942107 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5c564f7cc4-f5lb9_56d22a0c-06e1-45b3-a192-ae704c5c4bb1/maas-api/0.log" Sep 11 20:40:23.267241 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:23.267161 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-wjc6h_d9051b97-427f-482e-9d53-7163fc50cf95/postgres/0.log" Sep 11 20:40:24.709463 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:24.709429 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-gdgsc_34685262-d863-498c-9550-f9a0a20cef98/manager/0.log" Sep 11 20:40:25.168892 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:25.168856 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-pnz6j_d6aef9ef-d852-445e-9538-7e487cdbbf6f/discovery/0.log" Sep 11 20:40:25.393481 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:25.393451 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-567ff9dcd5-xpbxn_3c3d1d29-13cf-48b4-b5a5-4da836cc47cb/kube-auth-proxy/0.log" Sep 11 20:40:25.505659 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:25.505559 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-r787z_07f88bd9-1f8b-400f-bf08-e1a1b90c014f/istio-proxy/1.log" Sep 11 20:40:25.620386 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:25.620356 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-hvcj7_1b46934e-d75e-4dd9-8bdc-3329269b451b/payload-pre-processing/0.log" Sep 11 20:40:26.180913 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:26.180879 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-wxg77_9cfbc7f5-c1ef-4ea4-9087-5b619572d396/main/0.log" Sep 11 20:40:26.506307 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:26.506232 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-vq928_aad104bb-dc8a-48f5-9ff6-33c6e98102f7/main/0.log" Sep 11 20:40:26.614125 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:26.614097 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffpqcl_22cd130b-ace8-4d20-a769-46fd823ace55/main/0.log" Sep 11 20:40:31.717905 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:31.717872 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:40:31.778064 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:31.778037 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:40:34.475262 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:34.475229 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-q2dm2_1f86594f-112f-4e58-a2ca-bbd951c4d1af/global-pull-secret-syncer/0.log" Sep 11 20:40:34.552839 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:34.552811 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" Sep 11 20:40:34.661173 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:34.661141 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-205.ec2.internal_11ecaef2c363c5ae25ba196bd5dc1142/haproxy/0.log" Sep 11 20:40:39.627047 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:39.627012 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-gdgsc_34685262-d863-498c-9550-f9a0a20cef98/manager/0.log" Sep 11 20:40:41.122407 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:41.122382 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3ebf73f5-6a83-4e74-ab9a-28e881e2c57d/alertmanager/0.log" Sep 11 20:40:41.142246 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:41.142221 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3ebf73f5-6a83-4e74-ab9a-28e881e2c57d/config-reloader/0.log" Sep 11 20:40:41.164750 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:41.164718 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3ebf73f5-6a83-4e74-ab9a-28e881e2c57d/kube-rbac-proxy-web/0.log" Sep 11 20:40:41.182533 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:41.182513 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3ebf73f5-6a83-4e74-ab9a-28e881e2c57d/kube-rbac-proxy/0.log" Sep 11 20:40:41.205100 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:41.205076 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3ebf73f5-6a83-4e74-ab9a-28e881e2c57d/kube-rbac-proxy-metric/0.log" Sep 11 20:40:41.228474 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:41.228456 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3ebf73f5-6a83-4e74-ab9a-28e881e2c57d/prom-label-proxy/0.log" Sep 11 20:40:41.247354 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:41.247328 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3ebf73f5-6a83-4e74-ab9a-28e881e2c57d/init-config-reloader/0.log" Sep 11 20:40:41.439786 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:41.439757 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9x7w5_d2d5f666-660a-47aa-85c7-4ba6e485c395/node-exporter/0.log" Sep 11 20:40:41.456990 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:41.456962 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9x7w5_d2d5f666-660a-47aa-85c7-4ba6e485c395/kube-rbac-proxy/0.log" Sep 11 20:40:41.474689 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:41.474661 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9x7w5_d2d5f666-660a-47aa-85c7-4ba6e485c395/init-textfile/0.log" Sep 11 20:40:41.707176 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:41.707152 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d87454d6-3a5a-4a45-b925-fe4f4bd5d697/prometheus/0.log" Sep 11 20:40:41.725685 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:41.725659 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d87454d6-3a5a-4a45-b925-fe4f4bd5d697/config-reloader/0.log" Sep 11 20:40:41.747643 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:41.747618 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d87454d6-3a5a-4a45-b925-fe4f4bd5d697/thanos-sidecar/0.log" Sep 11 20:40:41.765256 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:41.765222 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d87454d6-3a5a-4a45-b925-fe4f4bd5d697/kube-rbac-proxy-web/0.log" Sep 11 20:40:41.783265 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:41.783232 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d87454d6-3a5a-4a45-b925-fe4f4bd5d697/kube-rbac-proxy/0.log" Sep 11 20:40:41.801822 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:41.801792 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d87454d6-3a5a-4a45-b925-fe4f4bd5d697/kube-rbac-proxy-thanos/0.log" Sep 11 20:40:41.824250 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:41.824228 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d87454d6-3a5a-4a45-b925-fe4f4bd5d697/init-config-reloader/0.log" Sep 11 20:40:41.856413 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:41.856386 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-xrnbx_e40d01fe-6f2e-4dce-95af-72a07ce53d56/prometheus-operator/0.log" Sep 11 20:40:41.884272 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:41.884244 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-xrnbx_e40d01fe-6f2e-4dce-95af-72a07ce53d56/kube-rbac-proxy/0.log" Sep 11 20:40:41.906138 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:41.906071 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-8g4nj_a3200c2f-fb6b-4a21-83b4-3c0e04f084ae/prometheus-operator-admission-webhook/0.log" Sep 11 20:40:41.942631 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:41.942602 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-fc5d4f5c9-bksvl_e9849d8f-4428-4c98-9aeb-f02cc00775d3/telemeter-client/0.log" Sep 11 20:40:41.966181 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:41.966152 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-fc5d4f5c9-bksvl_e9849d8f-4428-4c98-9aeb-f02cc00775d3/reload/0.log" Sep 11 20:40:41.990215 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:41.990190 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-fc5d4f5c9-bksvl_e9849d8f-4428-4c98-9aeb-f02cc00775d3/kube-rbac-proxy/0.log" Sep 11 20:40:42.023410 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:42.023386 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b5b4c8886-wf7z8_0d6de2e0-f1ba-443b-9624-7144ab50bf04/thanos-query/0.log" Sep 11 20:40:42.044289 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:42.044255 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b5b4c8886-wf7z8_0d6de2e0-f1ba-443b-9624-7144ab50bf04/kube-rbac-proxy-web/0.log" Sep 11 20:40:42.070765 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:42.070737 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b5b4c8886-wf7z8_0d6de2e0-f1ba-443b-9624-7144ab50bf04/kube-rbac-proxy/0.log" Sep 11 20:40:42.094511 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:42.094485 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b5b4c8886-wf7z8_0d6de2e0-f1ba-443b-9624-7144ab50bf04/prom-label-proxy/0.log" Sep 11 20:40:42.122048 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:42.122022 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b5b4c8886-wf7z8_0d6de2e0-f1ba-443b-9624-7144ab50bf04/kube-rbac-proxy-rules/0.log" Sep 11 20:40:42.141298 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:42.141265 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b5b4c8886-wf7z8_0d6de2e0-f1ba-443b-9624-7144ab50bf04/kube-rbac-proxy-metrics/0.log" Sep 11 20:40:44.477410 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:44.477382 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mjrrg/perf-node-gather-daemonset-g4p2s"] Sep 11 20:40:44.477824 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:44.477809 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62df1741-005c-49a4-8166-faf72dd73a39" containerName="maas-api" Sep 11 20:40:44.477824 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:44.477825 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="62df1741-005c-49a4-8166-faf72dd73a39" containerName="maas-api" Sep 11 20:40:44.477927 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:44.477918 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="62df1741-005c-49a4-8166-faf72dd73a39" containerName="maas-api" Sep 11 20:40:44.481336 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:44.481315 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mjrrg/perf-node-gather-daemonset-g4p2s" Sep 11 20:40:44.483268 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:44.483248 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mjrrg\"/\"kube-root-ca.crt\"" Sep 11 20:40:44.483361 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:44.483296 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mjrrg\"/\"default-dockercfg-t9mmc\"" Sep 11 20:40:44.483710 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:44.483695 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mjrrg\"/\"openshift-service-ca.crt\"" Sep 11 20:40:44.487477 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:44.487453 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f7c0f8b-3636-4da8-a5ce-ba65ee38524c-sys\") pod \"perf-node-gather-daemonset-g4p2s\" (UID: \"0f7c0f8b-3636-4da8-a5ce-ba65ee38524c\") " pod="openshift-must-gather-mjrrg/perf-node-gather-daemonset-g4p2s" Sep 11 20:40:44.487612 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:44.487519 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6md5\" (UniqueName: \"kubernetes.io/projected/0f7c0f8b-3636-4da8-a5ce-ba65ee38524c-kube-api-access-s6md5\") pod \"perf-node-gather-daemonset-g4p2s\" (UID: \"0f7c0f8b-3636-4da8-a5ce-ba65ee38524c\") " pod="openshift-must-gather-mjrrg/perf-node-gather-daemonset-g4p2s" Sep 11 20:40:44.487612 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:44.487596 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0f7c0f8b-3636-4da8-a5ce-ba65ee38524c-proc\") pod \"perf-node-gather-daemonset-g4p2s\" (UID: \"0f7c0f8b-3636-4da8-a5ce-ba65ee38524c\") " pod="openshift-must-gather-mjrrg/perf-node-gather-daemonset-g4p2s" Sep 11 20:40:44.487752 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:44.487654 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0f7c0f8b-3636-4da8-a5ce-ba65ee38524c-lib-modules\") pod \"perf-node-gather-daemonset-g4p2s\" (UID: \"0f7c0f8b-3636-4da8-a5ce-ba65ee38524c\") " pod="openshift-must-gather-mjrrg/perf-node-gather-daemonset-g4p2s" Sep 11 20:40:44.487752 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:44.487735 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0f7c0f8b-3636-4da8-a5ce-ba65ee38524c-podres\") pod \"perf-node-gather-daemonset-g4p2s\" (UID: \"0f7c0f8b-3636-4da8-a5ce-ba65ee38524c\") " pod="openshift-must-gather-mjrrg/perf-node-gather-daemonset-g4p2s" Sep 11 20:40:44.488495 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:44.488470 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mjrrg/perf-node-gather-daemonset-g4p2s"] Sep 11 20:40:44.588200 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:44.588149 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0f7c0f8b-3636-4da8-a5ce-ba65ee38524c-lib-modules\") pod \"perf-node-gather-daemonset-g4p2s\" (UID: \"0f7c0f8b-3636-4da8-a5ce-ba65ee38524c\") " pod="openshift-must-gather-mjrrg/perf-node-gather-daemonset-g4p2s" Sep 11 20:40:44.588200 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:44.588208 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0f7c0f8b-3636-4da8-a5ce-ba65ee38524c-podres\") pod \"perf-node-gather-daemonset-g4p2s\" (UID: \"0f7c0f8b-3636-4da8-a5ce-ba65ee38524c\") " pod="openshift-must-gather-mjrrg/perf-node-gather-daemonset-g4p2s" Sep 11 20:40:44.588474 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:44.588267 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f7c0f8b-3636-4da8-a5ce-ba65ee38524c-sys\") pod \"perf-node-gather-daemonset-g4p2s\" (UID: \"0f7c0f8b-3636-4da8-a5ce-ba65ee38524c\") " pod="openshift-must-gather-mjrrg/perf-node-gather-daemonset-g4p2s" Sep 11 20:40:44.588474 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:44.588323 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6md5\" (UniqueName: \"kubernetes.io/projected/0f7c0f8b-3636-4da8-a5ce-ba65ee38524c-kube-api-access-s6md5\") pod \"perf-node-gather-daemonset-g4p2s\" (UID: \"0f7c0f8b-3636-4da8-a5ce-ba65ee38524c\") " pod="openshift-must-gather-mjrrg/perf-node-gather-daemonset-g4p2s" Sep 11 20:40:44.588474 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:44.588344 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0f7c0f8b-3636-4da8-a5ce-ba65ee38524c-lib-modules\") pod \"perf-node-gather-daemonset-g4p2s\" (UID: \"0f7c0f8b-3636-4da8-a5ce-ba65ee38524c\") " pod="openshift-must-gather-mjrrg/perf-node-gather-daemonset-g4p2s" Sep 11 20:40:44.588474 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:44.588371 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0f7c0f8b-3636-4da8-a5ce-ba65ee38524c-proc\") pod \"perf-node-gather-daemonset-g4p2s\" (UID: \"0f7c0f8b-3636-4da8-a5ce-ba65ee38524c\") " pod="openshift-must-gather-mjrrg/perf-node-gather-daemonset-g4p2s" Sep 11 20:40:44.588474 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:44.588396 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0f7c0f8b-3636-4da8-a5ce-ba65ee38524c-podres\") pod \"perf-node-gather-daemonset-g4p2s\" (UID: \"0f7c0f8b-3636-4da8-a5ce-ba65ee38524c\") " pod="openshift-must-gather-mjrrg/perf-node-gather-daemonset-g4p2s" Sep 11 20:40:44.588474 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:44.588423 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f7c0f8b-3636-4da8-a5ce-ba65ee38524c-sys\") pod \"perf-node-gather-daemonset-g4p2s\" (UID: \"0f7c0f8b-3636-4da8-a5ce-ba65ee38524c\") " pod="openshift-must-gather-mjrrg/perf-node-gather-daemonset-g4p2s" Sep 11 20:40:44.588474 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:44.588440 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0f7c0f8b-3636-4da8-a5ce-ba65ee38524c-proc\") pod \"perf-node-gather-daemonset-g4p2s\" (UID: \"0f7c0f8b-3636-4da8-a5ce-ba65ee38524c\") " pod="openshift-must-gather-mjrrg/perf-node-gather-daemonset-g4p2s" Sep 11 20:40:44.595525 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:44.595500 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6md5\" (UniqueName: \"kubernetes.io/projected/0f7c0f8b-3636-4da8-a5ce-ba65ee38524c-kube-api-access-s6md5\") pod \"perf-node-gather-daemonset-g4p2s\" (UID: \"0f7c0f8b-3636-4da8-a5ce-ba65ee38524c\") " pod="openshift-must-gather-mjrrg/perf-node-gather-daemonset-g4p2s" Sep 11 20:40:44.791871 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:44.791782 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mjrrg/perf-node-gather-daemonset-g4p2s" Sep 11 20:40:44.848093 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:44.848069 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7c4d7c7587-68mg4_e5e14f90-ad2f-43f6-92c3-e05d52e97dab/console/0.log" Sep 11 20:40:45.064388 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:45.064324 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mjrrg/perf-node-gather-daemonset-g4p2s"] Sep 11 20:40:45.066688 ip-10-0-138-205 kubenswrapper[2546]: W0911 20:40:45.066661 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0f7c0f8b_3636_4da8_a5ce_ba65ee38524c.slice/crio-87742aefccfe18c11d4d07cccf232b4a14cde73dc59b6f352d8fb3aacd881457 WatchSource:0}: Error finding container 87742aefccfe18c11d4d07cccf232b4a14cde73dc59b6f352d8fb3aacd881457: Status 404 returned error can't find the container with id 87742aefccfe18c11d4d07cccf232b4a14cde73dc59b6f352d8fb3aacd881457 Sep 11 20:40:45.459206 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:45.459164 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mjrrg/perf-node-gather-daemonset-g4p2s" event={"ID":"0f7c0f8b-3636-4da8-a5ce-ba65ee38524c","Type":"ContainerStarted","Data":"4e24d8eec7977838dd9700d31d986a7e5b06df4340a774a315b92ef85d5d4dd6"} Sep 11 20:40:45.459396 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:45.459239 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mjrrg/perf-node-gather-daemonset-g4p2s" Sep 11 20:40:45.459396 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:45.459261 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mjrrg/perf-node-gather-daemonset-g4p2s" event={"ID":"0f7c0f8b-3636-4da8-a5ce-ba65ee38524c","Type":"ContainerStarted","Data":"87742aefccfe18c11d4d07cccf232b4a14cde73dc59b6f352d8fb3aacd881457"} Sep 11 20:40:45.474035 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:45.473986 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mjrrg/perf-node-gather-daemonset-g4p2s" podStartSLOduration=1.473971399 podStartE2EDuration="1.473971399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 20:40:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:40:45.472377765 +0000 UTC m=+2415.520428111" watchObservedRunningTime="2026-09-11 20:40:45.473971399 +0000 UTC m=+2415.522021742" Sep 11 20:40:46.644934 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:46.644902 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rvgwn_488ce64f-e146-4301-8f4f-a8ca95abf4ae/dns/0.log" Sep 11 20:40:46.662664 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:46.662637 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rvgwn_488ce64f-e146-4301-8f4f-a8ca95abf4ae/kube-rbac-proxy/0.log" Sep 11 20:40:46.684452 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:46.684421 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2m6cs_2133a2c0-f263-45aa-9d96-1819721a8c9b/dns-node-resolver/0.log" Sep 11 20:40:47.232218 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:47.232194 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-54fff_6e8ee8b7-3b67-4b9d-88fc-552fd655d11a/node-ca/0.log" Sep 11 20:40:48.535070 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:48.535042 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-pnz6j_d6aef9ef-d852-445e-9538-7e487cdbbf6f/discovery/0.log" Sep 11 20:40:48.575412 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:48.575381 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-567ff9dcd5-xpbxn_3c3d1d29-13cf-48b4-b5a5-4da836cc47cb/kube-auth-proxy/0.log" Sep 11 20:40:48.602404 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:48.602378 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-r787z_07f88bd9-1f8b-400f-bf08-e1a1b90c014f/istio-proxy/0.log" Sep 11 20:40:48.604824 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:48.604805 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-r787z_07f88bd9-1f8b-400f-bf08-e1a1b90c014f/istio-proxy/1.log" Sep 11 20:40:48.632343 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:48.632320 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-hvcj7_1b46934e-d75e-4dd9-8bdc-3329269b451b/payload-pre-processing/0.log" Sep 11 20:40:49.528865 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:49.528838 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-82bcx_1fdd17f8-61f0-45b2-a8d9-809c6d83ff03/serve-healthcheck-canary/0.log" Sep 11 20:40:50.063652 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:50.063621 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gxt9c_9ed8a847-72ef-440f-a4f2-dae8cdb83925/kube-rbac-proxy/0.log" Sep 11 20:40:50.080811 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:50.080781 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gxt9c_9ed8a847-72ef-440f-a4f2-dae8cdb83925/exporter/0.log" Sep 11 20:40:50.098680 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:50.098656 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gxt9c_9ed8a847-72ef-440f-a4f2-dae8cdb83925/extractor/0.log" Sep 11 20:40:51.473396 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:51.473366 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mjrrg/perf-node-gather-daemonset-g4p2s" Sep 11 20:40:52.308195 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:52.308153 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-xzdbn_7c00411d-ee5f-4b0f-8c45-2daa2426389d/manager/0.log" Sep 11 20:40:52.537248 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:52.537218 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-hs68x_9c8afe06-4c3c-41dd-9adb-f3cdc51ee68d/manager/0.log" Sep 11 20:40:52.582993 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:52.582871 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-57fc56fd56-srkrt_27ecd431-11ab-4ad3-9f75-4d79e7de99f7/manager/0.log" Sep 11 20:40:52.601382 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:52.601348 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-hmfh4_30b6578d-630f-4290-ad87-b335c2eef845/server/0.log" Sep 11 20:40:52.678048 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:52.678016 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-htbhw_0c85ece7-fbaf-4a92-99d5-e48a0170cd2d/manager/0.log" Sep 11 20:40:52.787388 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:52.787355 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5fdf48ff49-xthdd_be81dfc2-533a-441c-950c-83a3c63a4c6c/manager/0.log" Sep 11 20:40:52.802308 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:52.802284 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5fdf48ff49-xthdd_be81dfc2-533a-441c-950c-83a3c63a4c6c/copy-manifests/0.log" Sep 11 20:40:54.156167 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:54.156138 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-5t8mh_26729964-bf2e-4b51-9680-9efbfb654783/openshift-lws-operator/0.log" Sep 11 20:40:54.712467 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:54.712435 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-pnz6j_d6aef9ef-d852-445e-9538-7e487cdbbf6f/discovery/0.log" Sep 11 20:40:54.741477 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:54.741439 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-567ff9dcd5-xpbxn_3c3d1d29-13cf-48b4-b5a5-4da836cc47cb/kube-auth-proxy/0.log" Sep 11 20:40:54.760214 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:54.760174 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-r787z_07f88bd9-1f8b-400f-bf08-e1a1b90c014f/istio-proxy/0.log" Sep 11 20:40:54.762715 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:54.762695 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-r787z_07f88bd9-1f8b-400f-bf08-e1a1b90c014f/istio-proxy/1.log" Sep 11 20:40:54.783542 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:54.783520 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-hvcj7_1b46934e-d75e-4dd9-8bdc-3329269b451b/payload-pre-processing/0.log" Sep 11 20:40:56.791803 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:56.791725 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-pnz6j_d6aef9ef-d852-445e-9538-7e487cdbbf6f/discovery/0.log" Sep 11 20:40:56.825305 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:56.825281 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-567ff9dcd5-xpbxn_3c3d1d29-13cf-48b4-b5a5-4da836cc47cb/kube-auth-proxy/0.log" Sep 11 20:40:56.845909 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:56.845882 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-r787z_07f88bd9-1f8b-400f-bf08-e1a1b90c014f/istio-proxy/0.log" Sep 11 20:40:56.848629 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:56.848604 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-r787z_07f88bd9-1f8b-400f-bf08-e1a1b90c014f/istio-proxy/1.log" Sep 11 20:40:56.868493 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:56.868470 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-hvcj7_1b46934e-d75e-4dd9-8bdc-3329269b451b/payload-pre-processing/0.log" Sep 11 20:40:56.934651 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:56.934627 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:40:58.139327 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:40:58.139294 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jdhk4_1ab887bb-e9fb-4ad3-9178-a35ed6fecfed/konnectivity-agent/0.log" limit=1024 Sep 11 20:41:00.552542 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:41:00.552512 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8kcqg_171c51ac-8f0e-426f-973d-5a596e73d7d1/kube-multus/0.log" Sep 11 20:41:00.745738 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:41:00.745711 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ddmz9_52c72e1a-d061-4d3c-922b-b572a6496fed/kube-multus-additional-cni-plugins/0.log" Sep 11 20:41:00.762570 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:41:00.762540 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ddmz9_52c72e1a-d061-4d3c-922b-b572a6496fed/egress-router-binary-copy/0.log" Sep 11 20:41:00.781607 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:41:00.781571 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ddmz9_52c72e1a-d061-4d3c-922b-b572a6496fed/cni-plugins/0.log" Sep 11 20:41:00.798297 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:41:00.798273 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ddmz9_52c72e1a-d061-4d3c-922b-b572a6496fed/bond-cni-plugin/0.log" Sep 11 20:41:00.816775 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:41:00.816724 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ddmz9_52c72e1a-d061-4d3c-922b-b572a6496fed/routeoverride-cni/0.log" Sep 11 20:41:00.835651 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:41:00.835629 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ddmz9_52c72e1a-d061-4d3c-922b-b572a6496fed/whereabouts-cni-bincopy/0.log" Sep 11 20:41:00.852504 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:41:00.852480 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ddmz9_52c72e1a-d061-4d3c-922b-b572a6496fed/whereabouts-cni/0.log" Sep 11 20:41:01.094407 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:41:01.094380 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zpl5q_ce96d552-27b5-4912-86b0-38f0f46d0002/network-metrics-daemon/0.log" Sep 11 20:41:01.108796 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:41:01.108769 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zpl5q_ce96d552-27b5-4912-86b0-38f0f46d0002/kube-rbac-proxy/0.log" Sep 11 20:41:02.048184 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:41:02.048154 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pgnnf_b6b30385-b0e1-4ab5-916a-57992046f196/ovn-controller/0.log" Sep 11 20:41:02.062675 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:41:02.062649 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pgnnf_b6b30385-b0e1-4ab5-916a-57992046f196/ovn-acl-logging/0.log" Sep 11 20:41:02.081094 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:41:02.081076 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pgnnf_b6b30385-b0e1-4ab5-916a-57992046f196/kube-rbac-proxy-node/0.log" Sep 11 20:41:02.099263 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:41:02.099245 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pgnnf_b6b30385-b0e1-4ab5-916a-57992046f196/kube-rbac-proxy-ovn-metrics/0.log" Sep 11 20:41:02.116062 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:41:02.116045 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pgnnf_b6b30385-b0e1-4ab5-916a-57992046f196/northd/0.log" Sep 11 20:41:02.133560 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:41:02.133541 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pgnnf_b6b30385-b0e1-4ab5-916a-57992046f196/nbdb/0.log" Sep 11 20:41:02.150663 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:41:02.150642 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pgnnf_b6b30385-b0e1-4ab5-916a-57992046f196/sbdb/0.log" Sep 11 20:41:02.203629 ip-10-0-138-205 kubenswrapper[2546]: I0911 20:41:02.203609 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pgnnf_b6b30385-b0e1-4ab5-916a-57992046f196/ovnkube-controller/0.log"